EncouRAGe: Evaluating RAG Local, Fast, and Reliable

2025

Read EncouRAGe: Evaluating RAG Local, Fast, and Reliable (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04696.txt.

Introduction

Jan Strich and three colleagues introduce EncouRAGe, a Python framework for testing retrieval-augmented generation. Such systems pair language models with searched documents, but setups vary wildly between labs. EncouRAGe breaks the pipeline into five modular parts that researchers can swap.

The parts cover type definitions, system building, inference, the vector store, and metrics. Local deployment and repeatable results are design goals throughout. Evaluation spans several benchmarks with twenty-five thousand question pairs and over fifty-one thousand documents. Their results carry a warning: retrieval systems still trail some baselines, so careful local testing remains essential.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EncouRAGe: Evaluating RAG Local, Fast, and Reliable Abstract We introduce EncouRAGe , a comprehensive Python framework designed to streamline the development and evaluation of Retrieval-Augmented Generation (RAG) systems using Large Language Models (LLMs) and Embedding Models. EncouRAGe comprises five modular and extensible components: Type Manifest , RAG Factory , Inference , Vector Store , and Metrics , facilitating flexible experimentation and extensible development. The framework emphasizes scientific reproducibility , diverse evaluation metrics , and local deployment , enabling researchers to efficiently assess datasets within RAG workflows. This paper presents implementation details and an extensive evaluation across multiple benchmark datasets, including 25k QA pairs and over 51k documents . Our results show that RAG still underperforms compared to the Oracle Context , while Hybrid BM25 consistently achieves the best results across all four datasets. We further examine the effects of reranking, observing only marginal performance improvements accompanied by higher response latency. Keywords: Retrieval-Augmented Generation, Evaluation, Library EncouRAGe: Evaluating RAG Local, Fast, and Reliable Jan Strich, Adeline Scharfenberg, Chris Biemann, Martin Semmann Universität Hamburg Abstract content 1. Introduction Retrieval-Augmented Generation (RAG) ( Lewis et al. 2020 ; Huang and Huang 2024 ; Nikishina et al. 2025 ) has emerged as a dominant approach for enhancing large language models (LLMs) with external knowledge sources. By combining retrieval and generation, RAG systems can provide more factual, contextually grounded, and up-to-date responses ( Shuster et al. 2021 ; Sahoo et al. 2024 ) . However, the rapid development of RAG architectures ( Huang and Huang 2024 ; Sarthi et al. 2024 ) , methods ( Gao et al. 2021 ; Gao et al. 2023 ) , and LLM-as-a-Judge metrics ( Es et al. 2024 ) has created a need for an implementation-focused RAG evaluation library to create a reliable way to compare performance on domain-specific datasets. Assessing a RAG system for a specific dataset requires analyzing the performance of both the retriever and generator, as well as their interaction. It is particularly challenging because LLM outputs can be lengthy, complex, and ambiguous, making it difficult to define clear correctness or objectively measure factual accuracy ( Ru et al. 2024 ) . Recent RAG evaluation frameworks, such as RAGAS ( Saad-Falcon et al. 2024 ) , RAGChecker ( Ru et al. 2024 ) , and TruLens ( Snowflake Inc. 2024 ) , have begun addressing these challenges by providing libraries to jointly analyze retrieval and generation quality, reflecting the interdependence between the two components. Instead of relying solely on traditional metrics like BLEU ( Papineni et al. 2002 ) or ROUGE ( Lin 2004 ) , they use LLM-based or embedding-based measures to directly evaluate factual correctness, relevance, and consistency of the generated text ( Es et al. 2024 ) . Additionally, they often provide enhanced UI and monitoring features for RAG in production environments. Despite recent advances, a lack of the following features in state-of-the-art (SOTA) frameworks remains, hindering the effective comparison of RAG methods for domain-specific datasets: (1) Lack of comparability of scientific methods. Existing libraries in this field focus solely on implementing metrics, with no standardized implementation of SOTA RAG methods. This heterogeneity hinders meaningful and reproducible comparisons across different RAG approaches. (2) Limited flexibility of LLM-as-a-judge metrics. LLM-as-a-judge metrics are sensitive to domain and dataset, and must be flexible to be effective. Therefore, custom prompts for individually designing evaluations are crucial and are not currently available in existing frameworks. (3) Primarily cloud-oriented, with limited flexibility for integrating additional metrics or methods. Many existing frameworks are designed with a narrow focus on specific local setups and lack modularity for broader integration. A flexible, plug-and-play architecture is essential to support new RAG methods and additional new metrics. Figure 1: System overview of the EncouRAGe Python library. Input data in any format must be transformed to fit the Type manifest . The RAG Factory provides various RAG methods, while the Metrics component implements all evaluation metrics. Inference can be executed locally or via cloud providers using the OpenAI SDK, and supported Vector stores include Chroma and Qdrant. The output fits popular monitoring systems. To tackle these challenges, we present EncouRAGe , an open-source Python library for comprehensive, reproducible, and extensible RAG evaluation (Figure 1 ). It integrates any dataset into a parsable, object-oriented structure for effective and traceable workflows. EncouRAGe includes 10 RAG methods, manages LLM and embedding inference and provides metrics in three categories: 1. Generator Metrics : Evaluating the performance end-to-end with classic NLP metrics (ROUGE, BLEU, etc.). 2. Retrieval Metrics : Evaluating the effectiveness of the retriever in finding relevant information from the knowledge base. 3. LLM-as-a-Judge : Evaluating the effectiveness of the whole RAG Pipeline enhanced by custom LLM prompts. To address the rapid pace and volume of new RAG and LLM research, reproducing and comparing methods has become increasingly time-consuming. With EncouRAGe, evaluating new approaches or verifying others claims becomes straightforward and extensible. The framework enables quick benchmarking across diverse methods and datasets with minimal manual effort, requiring mainly GPU time rather than complex reimplementation. We demonstrate this through experiments comparing multiple RAG methods on four popular QA datasets. The datasets and code are open source and anonymized for submission. 1 1 1 https://anonymous.4open.science/r/encourage-B501/ The main contributions of this paper are as follows: • We present EncouRAGe , a novel RAG evaluation framework that consists of 10 SOTA RAG methods , an object-oriented type manifest , and over 20 metrics to get transparent and reliable results. • We conduct a comparison of RAG methods using EncouRAGe on four datasets with 25k QA pairs and 50k documents , demonstrating the effectiveness and usefulness of the framework. • We further present one ablation study, showing the effectiveness of Reranker ratio for two cross-encoder models and a subset of the four datasets. 2. Related Work 2.1. Retrieval-Augmented Generation LLMs excel at text generation but face challenges such as outdated knowledge and hallucinations ( Shuster et al. 2021 ; Huang and Huang 2024 ) . RAG addresses these issues by incorporating external knowledge, improving accuracy and factuality ( Lewis et al. 2020 ; Sarthi et al. 2024 ) . This approach is critical in domains such as law Cui et al. 2024 , medicine Xiong et al. 2024 ; Chen et al. 2025b , and finance Chen et al. 2025b , where precision is crucial. RAG systems have achieved strong results in tasks such as open-domain question answering Kim and Lee 2024 , code generation Wang et al. 2025 , and dialogue Chen et al. 2025a . They have also been successfully adopted in real-world applications, including LlamaIndex Liu 2022 . 2.2. Evaluation of RAG Evaluation RAG systems can be categorized into three domains: tools for lightweight and dataset-based evaluation, monitoring frameworks for production applications, and general-purpose evaluation frameworks for LLMs. (1) Lightweight and Dataset-Based Evaluation Tools. RAGAS ( Es et al. 2024 ) and RAGChecker ( Ru et al. 2024 ) provide systematic metrics for assessing RAG pipelines. RAGAS was the first library to provide LLM-as-a-Judge metrics and offers additional classic NLP evaluation metrics. In contrast, RAGChecker employs a modular approach that decomposes RAG performance into retrieval and generation components, allowing detailed error attribution and interpretability. But both lack RAG method implementation and

Read the complete text