Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering

Sai Shridhar Balamurali, Lu Cheng · 2025

Read Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering by Sai Shridhar Balamurali, Lu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07659.txt.

Introduction

Sai Shridhar Balamurali and Lu Cheng revive an old tool for judging language model answers. Lexical scores miss meaning, while using a giant model as judge costs heavily. Natural language inference, which checks whether one text entails another, offers a cheap middle path.

Adding a simple lexical-match flag lifts inference scoring to match a top commercial judge on long-form questions, with far fewer parameters. A new benchmark of three thousand human-rated samples across five datasets and five models tests alignment rigorously. Cheap metrics that track human judgment let teams evaluate more answers, more often, without renting a giant judge for every check.

Catalog record

CREATORSai Shridhar Balamurali, Lu Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07659

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering Sai Shridhar Balamurali, Lu Cheng Abstract Evaluating answers from state-of-the-art large language models (LLMs) is challenging: lexical metrics miss semantic nuances, whereas “LLM-as-Judge” scoring is computationally expensive. We re-evaluate a lightweight alternative—off-the-shelf Natural Language Inference (NLI) scoring augmented by a simple lexical-match flag and find that this decades-old technique matches GPT-4o’s accuracy (89.9%) on long-form QA, while requiring orders-of-magnitude fewer parameters. To test human alignment of these metrics rigorously, we introduce DIVER-QA, a new 3000 sample human annotated benchmark spanning five QA datasets and five candidate LLMs. Our results highlight that inexpensive NLI-based evaluation remains competitive and offer DIVER-QA as an open resource for future metric research. 0 0 footnotetext: We acknowledge the use of AI assistants (GPT-4o/Claude 3.7) for editing assistance and improving clarity of expression in portions of this manuscript. All research content, analyses, and scientific contributions were conceived and developed by the authors. 1 Introduction The rapid advancement of large language models (LLMs) like GPT 6 ; 2 , LLaMA 30 ; 9 , and Gemini 29 ; 27 necessitates robust evaluation methodologies. Question Answering (QA) benchmarks like MMLU 11 , MMLU-pro 32 , and HotpotQA 34 have become standard for assessing model capabilities, but present evaluation challenges. While MMLU relies on multiple-choice questions that can introduce biases 19 , HotpotQA expects concise answers that don’t accommodate the verbose outputs modern LLMs often generate. This means that if a model generates the correct answer with different wording it will be considered wrong and could also create a disparity between model classes. Larger models can follow strict instructions to produce concise answers, but smaller models—despite possessing considerable knowledge—often struggle when constrained to restrictive formats. When allowed to generate explanatory responses, these models frequently demonstrate correct reasoning, as shown in this HotpotQA example using Mistral-7B 14 : Q: Were Scott Derrickson and Ed Wood of the same nationality? Reference: yes Mistral-7B response: Scott Derrickson is an American Director, while Ed Wood was a American filmmaker. Both are of the same nationality. Current evaluation approaches include: (1) traditional reference-based metrics like Exact Match, which lack semantic understanding; (2) neural model-based metrics 35 ; 28 ; 7 requiring task-specific training; and (3) computationally expensive "LLM-as-Judge" approaches 36 ; 21 . None effectively balance accuracy with efficiency when evaluating verbose outputs against reference answers, with each approach introducing potential biases 36 ; 12 . In this paper, we revisit Natural Language Inference (NLI) as a solution to this problem and also experiment augmenting a general NLI model with lexical matching to address substring matching weaknesses. Our weighted logistic regression framework achieves performance comparable to state-of-the-art models while remaining computationally efficient. Our contributions include: • A diverse QA dataset for testing metrics across modern instruction-tuned LLMs. • Evidence that NLI models can evaluate long-form QA responses nearly as well as GPT-4o (89.9% accuracy) with significantly lower costs. • Empirical proof of our approach’s generalizability and cost-effectiveness across various QA tasks. 2 Background and Related Works Traditional lexical metrics for QA, e.g., Exact Match (EM) and F1 score, are inadequate to evaluate contemporary LLMs. These LLMs often generate verbose, instruction-following responses where correct answers are embedded within broader context, leading to unfairly low scores on metrics designed for concise extractions 35 ; 28 ; 7 ; 27 ; 19 . This inadequacy spurred the development of neural metrics like BERTScore 35 , BLEURT 28 , which leveraged contextual embeddings to better capture semantic similarity, and BEM 7 which used semantic equivalence classification. While an improvement over purely lexical methods, these early neural approaches still exhibited limitations in discerning nuanced semantic equivalence and achieved only modest correlation with human judgment when answers were phrased very differently from references 28 . More recently, the use of LLMs themselves as evaluators (e.g., "LLM-as-judge" or for direct reference-based assessment) has become state-of-the-art, demonstrating superior semantic understanding and correlation with human ratings 17 ; 18 ; 36 . However, this approach carries a significant drawback: LLM-based evaluators, whether proprietary or open-source, are computationally intensive and prohibitively expensive for many researchers, hindering widespread adoption and reproducibility 12 . Furthermore, these LLM judges can introduce their own biases, such as verbosity or positional preferences 36 . This landscape reveals a critical gap: a need for evaluation metrics that can approach the semantic acuity of powerful LLM evaluators without their substantial computational and financial overhead. Our work addresses this by revisiting and enhancing an efficient neural approach to offer a robust, cost-effective alternative for evaluating modern LLM outputs. 3 Methodology Our research revisits NLI as a foundation for evaluating LLM-generated answers against reference responses. While NLI has been previously explored for QA evaluation 10 , we reexamine its effectiveness specifically for verbose outputs from instruction-tuned models and enhance it with complementary signals. 3.1 NLI-based Metric NLI offers a particularly suitable framework for QA evaluation because it directly addresses the core question: does the generated answer entail the reference answer? This formulation naturally handles paraphrasing, additional context, and varied expression—all common in modern LLM outputs. For our NLI model, we utilize DeBERTa-v3-large-NLI 20 , a model specifically built as a universal classifier using NLI. This model was trained on 33 datasets with 389 diverse classes, encompassing 885,242 NLI hypothesis-premise pairs from multiple sources including MultiNLI 33 , Fever-NLI 23 , Adversarial-NLI (ANLI) 24 ), LingNLI 25 , and WANLI 22 . The authors demonstrated that NLI can serve as a universal classification task following similar principles as instruction fine-tuning of generative LLMs, making it particularly well-suited for zero-shot classification tasks like QA evaluation. We selected this model specifically for its generalized zero-shot performance, which enables researchers to use it "out of the box" without requiring additional fine-tuning for specific QA datasets. This choice aligns with our goal of providing computationally efficient evaluation methods that remain accessible to researchers with limited resources. Given a question q , a candidate answer a and a reference answer r , we format inputs as "[CLS] question: q answer: a [SEP] question: q ground truth: r [SEP]" to provide the model with full context, taking the entailment probability as our semantic score. Entailment probability provides the most informative measure of semantic equivalence, as contradiction merely indicates opposing propositions without confirming similarity, while neutrality signifies an absence of inferential relationship. Within the tripartite probability distribution, entailment uniquely quantifies the degree to which one statement logically implies another. Figure 1: Framework for the NLI+lex model. 3.2 NLI Augmented with Lexical Equivalence While NLI excels at capturing semantic equivalence ( se(q,a,r) ), it can occasionally falter with precise factual assessments where an answer is explicitly stated but embedded within verbose context. To address this, we augment NLI with a complementary lexical matching signal ( lm(a,r) ), a simple binary function indicating if the reference r appears as a substring in the answer a . These two signals are linearly combined: z=w_{1}\cdot\texttt{se(q, a, r)}+w_{2}\cdot\texttt{lm(a, r)}. The weights ( w_{1} , w_{2} ) are learned through logistic regression, yielding a probability of correctness, P(\texttt{correct})=1/(1+e^{-z}) . Responses are then classified as correct (1) or incorrect (0) using a 0.5 threshold (Figure 1 . We can calibrate these weights using any QA

Read the complete text