Towards Lighter and Robust Evaluation for Retrieval Augmented Generation

Alex-Răzvan Ispas, Charles-Élie Simon, Fabien Caspani, Vincent Guigue · 2025

Read Towards Lighter and Robust Evaluation for Retrieval Augmented Generation by Alex-Răzvan Ispas, Charles-Élie Simon, Fabien Caspani, Vincent Guigue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16161.txt.

Introduction

Systems that answer questions by first looking up sources can still make up facts, and checking their answers often relies on costly closed AI models. Alex-Răzvan Ispas and co-authors offer a lighter path in a 2025 paper. They test whether small, open models can grade answers for correctness and faithfulness to the sources.

Their approach squeezes small language models so they run cheaply, then has them give each answer a graded score rather than a yes or no. Those scores let users question how much to trust a grading call. The team also builds a new summary metric from the scores as a stand-in for human judgment. Open grading tools could make answer checking cheaper and easier to inspect.

Catalog record

CREATORAlex-Răzvan Ispas, Charles-Élie Simon, Fabien Caspani, Vincent Guigue
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Lighter and Robust Evaluation for Retrieval Augmented Generation Alex-Răzvan Ispas, Charles-Élie Simon, Fabien Caspani, Vincent Guigue Abstract Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions’ reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment. We also provide a script for the corrected dataset which is available at: GitHub Repository 1 Introduction Large Language Models (LLMs) have advanced the field of Natural Language Processing (NLP) in recent years Achiam et al. 2023 ; Touvron et al. 2023 ; Jiang et al. 2024 . However, some questions require information outside the knowledge scope of the model. Therefore, Retrieval Augmented Generation (RAG) Lewis et al. 2020 was proposed to enhance the quality of the answers for questions by retrieving information from a relevant knowledge base. RAG reliability remains a critical concern, particularly due to hallucinations in the generated answers. While much effort has been dedicated to improving model accuracy, a structured evaluation framework that explicitly addresses hallucination detection is still needed. In general, we want to assess the quality of an LLM answer by comparing it to a ground truth. Previous approaches used token-based metrics such as Exact Matching (EM) Rajpurkar et al. 2016 , which are fast to compute but not robust to semantic variations. Human evaluators are considered the most consistent approach for open evaluation, but they require a lot of resources Islam et al. 2023 . Information extraction approaches can be used to compare the facts in the generated response with a ground truth Jayanthi et al. 2021 , but such systems are quite domain-sensitive, and the metric may not be very robust to domain changes. NLI approaches Liu et al. 2021 can be used to validate statements, but typically in the context of simple sentences or using a combinatorial approach. LLM evaluators have recently risen in popularity, and they have been compared with human evaluators regarding agreement on different benchmarks such as MT-bench Zheng et al. 2024 . Factual few-shot prompting evaluation methods Es et al. 2023 ; Zhang et al. 2023 ; Manakul et al. 2023 can check the level of truth of each answer statement while avoiding the search complexity of a knowledge graph Sansford et al. 2024 . While factual evaluation exhibits progress, most proposed methods rely on enterprise models such as GPT4 Es et al. 2023 ; Manakul et al. 2023 , making the reproduction of experiments challenging due to financial and ethical considerations, such as the protection of sensitive data. Since numerous open-source lightweight LLMs offer comparable performance and are more accessible with quantization, we explore their capabilities with respect to enterprise models in terms of human agreement. In the meantime, we look for interpretability by providing an analysis at the statement level for each answer. Our contributions are as follows: • Proposing a reproducible framework for evaluating RAG-generated answers based on quantized models. • Defining a transparent and interpretable metric based on the decomposition and annotation of the response in terms of statements. • Providing the community with a corrected dataset for RAG evaluation. • Analyze the behaviour of the continuous metric to build a new alignment score with human judgment based on AUC. 2 Related Work Previous research on Question-Answering (QA), such as SQuAD Rajpurkar et al. 2016 , relied on deterministic approaches like exact matching (EM) and F1-score. The main advantages of deterministic metrics are that they guarantee correctness, are interpretable, and can be easily generalised. Although traditional metrics like BLEU Papineni et al. 2002 and ROUGE Lin 2004 perform well for specific tasks such as translation and summarization, they are not robust to task variations. Additionally, metrics that rely on token overlap cannot capture the semantic similarities between two answers. The standard method of evaluating text quality is through human evaluation. Human annotators have the advantage of understanding the context and human preferences much better. However, this method can introduce human bias, and the annotation process requires a significant amount of time and budget. Large Language Models used as evaluators are designed to act as a bridge between the previous approaches. They are capable of handling a wider range of open-ended NLP tasks that are in line with human preferences and that are more difficult to evaluate using traditional metrics Zheng et al. 2024 . The analysis conducted by Chiang & Lee 2023 highlights that LLMs can consistently assess text quality, like humans, when provided with the same instructions and examples. Unlike human evaluation, LLMs used as judges are a scalable alternative that minimizes the need for human involvement during the evaluation process and provides faster model iterations. However, positional bias makes the comparison of the generated answers challenging Wang et al. 2023 . According to Zheng et al. 2024 , there are three different ways of structuring the prompts. In pairwise comparison , the model receives a question and two answers and is asked to choose the better one. For pointwise scoring , the model is given a single answer and is asked to assign a score Manakul et al. 2023 . Reference-guided scoring provides the judge with a reference solution in addition to the question and the answer Es et al. 2023 . All of them can enhance their reasoning precision by applying chain-of-thought Wei et al. 2022 . In the case of RAG evaluation, Es et al. 2023 tried to overcome positional bias Wang et al. 2023 by using factual evaluation Manakul et al. 2023 , increasing the faithfulness accuracy of GPT3.5 from 0.72% to 0.95%. Other research Adlakha et al. 2023 compared the deterministic metrics and the LLM evaluators by computing the correlation with the human evaluators. They emphasised that bag-of-tokens recall is highly correlated with humans when the verbose information of the answer is ignored during the evaluation. At the same time, contextual embedding transformers such as BERTScore (BertS) have a lower correlation when they are used to calculate the recall Zhang et al. 2019 . Overall, GPT4 remains one of the highest-rated evaluators in most of the previous studies Zheng et al. 2024 . 3 Methodology Figure 1: Evaluation pipeline for answer correctness. First, the simplifier extracts the statements of the answer and the ones of the ground truth. Afterwards, the evaluator labels the statements according to the definitions. Finally, the parser extracts the labelled statements and calculates the metric. 3.1 Factual Evaluation Following the evaluation methods emphasized by Manakul et al. 2023 and Es et al. 2023 , we will evaluate

Read the complete text