Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge

Chase Fensore † \dagger , Kaustubh Dhole † \dagger , Joyce Ho ∗ , Eugene Agichtein ∗ · 2025

Read Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge by Chase Fensore † \dagger , Kaustubh Dhole † \dagger , Joyce Ho ∗ , Eugene Agichtein ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22644.txt.

Introduction

Chase Fensore and co-authors report their 2025 entry to the LiveRAG challenge, a contest for search backed question answering systems. Such systems fetch documents, then write answers that should be relevant and faithful to what was fetched. The challenge used a large web corpus and test sets made on the fly to limit memorized answers.

Their final system mixed classic keyword search with dense embedding search, then wrote answers with a mid sized open model. Tests on made up questions showed that adding a neural re ranking step lifted accuracy a lot but slowed answers to a crawl. Prompt tuning raised likeness scores yet never refused a question, raising overconfidence worries. The submitted system, without re ranking, placed high for faithfulness among dozens of teams. Matching question wording to document wording proved the strongest driver of success.

Catalog record

CREATORChase Fensore † \dagger , Kaustubh Dhole † \dagger , Joyce Ho ∗ , Eugene Agichtein ∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge Chase Fensore † \dagger , Kaustubh Dhole † \dagger , Joyce Ho ∗ , Eugene Agichtein ∗ Abstract . We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final hybrid approach combines sparse (BM25) and dense (E5) retrieval methods and then aims to generate relevant and faithful answers with Falcon3-10B-Instruct. Through systematic evaluation on 200 synthetic questions generated with DataMorgana across 64 unique question-user combinations, we demonstrate that neural re-ranking with RankLLaMA improves MAP from 0.523 to 0.797 (52% relative improvement) but introduces prohibitive computational costs (84s vs 1.74s per question). While DSPy-optimized prompting strategies achieved higher semantic similarity (0.771 vs 0.668), their 0% refusal rates raised concerns about over-confidence and generalizability. Our submitted hybrid system without re-ranking achieved 4 th place in faithfulness and 11 th place in correctness among 25 teams. Analysis across question categories reveals that vocabulary alignment between questions and documents was the strongest predictor of performance on our development set, with document-similar phrasing improving cosine similarity from 0.562 to 0.762. 1. Introduction The LiveRAG Challenge 2025 presents a unique evaluation framework for retrieval augmented generation (RAG) systems using dynamic test sets that capture the evolving nature of real-world information needs. Unlike traditional static benchmarks, this challenge evaluates systems on their ability to handle diverse question types, user expertise levels, and rapidly changing information landscapes using the FineWeb-10BT corpus ( Penedo et al. 2024 ) . Our approach focuses on developing a hybrid retrieval system that combines the complementary strengths of sparse and dense retrieval methods. Sparse retrieval excels at exact term matching and handling specific factual queries, while dense retrieval captures semantic similarity and contextual understanding. By combining these approaches and exploring document reformulation, neural re-ranking, and prompt optimization, we ultimately aimed to maximize correctness and faithfulness of our RAG system’s responses on the unseen LiveRAG Challenge Day test set. The key contributions of our work include: (1) a systematic evaluation of hybrid retrieval strategies on synthetic dataset we generated with the DataMorgana tool ( Filice et al. 2025 ) that mirrors real-world QA diversity, (2) analysis of performance across different question and user categories as defined by DataMorgana, (3) demonstration of the effectiveness of neural re-ranking in improving RAG system performance, which also comes at high computational cost, and (4) insights into trade-offs between development set optimization and real-world robustness through DSPy ( Khattab et al. 2023 ) prompt optimization experiments. This paper is organized as follows: In Section 2 , we first describe our retrieval and RAG pipeline in detail. In Section 3 , we demonstrate our experiments and discuss our results. 2. Methods We now describe our baseline experiments carried out to inform the development of our final RAG system. 2.1. Data Generation We utilized DataMorgana to create a development QA dataset (n=200) that reflects the diversity expected in real-world RAG applications. Our synthetic dataset incorporated multiple question categories and user categorizations, to simulate up to 64 unique combinations of RAG QA settings. Question Categories : We used four question categorizations consistent with those introduced by DataMorgana: ( Filice et al. 2025 ) (1) Question Factuality : Balancing factoid questions (50%) seeking specific information with open-ended questions (50%) encouraging detailed responses (2) Question Premise : Including direct questions (50%) without user context and premise-based questions (50%) where users provide relevant background (3) Question Phrasing : Covering concise natural questions (25%), verbose natural questions (25%), short search queries (25%), and long search queries (25%) (4) Question Linguistic Variation : Distinguishing between questions using document-similar terminology (50%) and document-distant phrasing (50%) User Categories : We focused on User Expertise categories, with expert users (80%) asking complex questions and novice users (20%) asking basic questions. 2.2. Retrieval Architecture Documents were first split into 512 token chunks with a sentence-aware splitter. For the initial stage of retrieval, we explored three retrieval strategies: (1) Sparse Retrieval : OpenSearch BM25 based index. k=30 documents (2) Dense Retrieval : Pinecone with E5 embeddings ( intfloat/ e5-base-v2 , size 768) and cosine similarity for retrieval. k=30 documents (3) Hybrid Retrieval : Combined sparse and dense retrieval (k=30 each), selecting top 10 documents based on normalized score fusion ( Lee 1995 ) Document Reformulation We explored doc2query-enhanced sparse indexing ( Nogueira et al. 2019 ) , where documents were augmented with generated questions to improve retrieval coverage. We created a custom doc2query-enhanced sparse BM25 index, using 512 token chunks similar to the OpenSearch BM25 index. When using this index, the top 30 documents were retrieved and included in the final prompt. However, this doc2query approach showed limited effectiveness in our experiments. Generative Re-ranking We employed RankLLaMA-7B ( castorini/rankllama -v1-7b-lora-passage ) ( Ma et al. 2024 ) for pointwise re-ranking of the top k retrieved passages — this means that the a given retrieved passage is assigned a score by the generative re-ranker independently of all other retrieved passages. Pointwise re-rankers have shown improved retrieval effectiveness in many tasks and are more robust compared to other approaches like listwise. To do this, the re-ranker uses the prompt structure from the pyterrier_genrank package ( Dhole 2024 ) , where each prompt contains exactly one query and one passage (Figure 1 ). Query: {query} Passage: {passage} Figure 1. Pointwise Re-ranking Prompt 2.3. Answer Generation We used the Falcon3-10B-Instruct ( Team 2024 ) model for answer generation with temperature=0.6 and top_p=0.9. We performed manual prompt engineering and arrived at a base prompt aimed to emphasize conciseness and faithfulness (Figure 2 ). """You are an AI assistant tasked with answering questions based on the provided information. Information: context as k chunks/documents Question: query Answer the question based only on the provided information. Keep the answer concise, limited to 200 tokens. If the information doesn’t contain the answer, say "I don’t have enough information to answer this question." Answer:""" Figure 2. Answer Generation Prompt Few-shot and Chain-of-Thought (CoT) approaches were also explored via DSPy ( Khattab et al. 2023 ) . For few-shot prompting using DSPy, we used a 160/40 train/validation split of our synthetic dataset. Answers were scored during prompt optimization with question similarity measured by MiniLM. 1 1 1 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 MIPROv2 and BootstrapFewShot optimizers were studied for CoT and few-shot respectively. For CoT prompting, we explored MIPROv2 to optimize the reasoning instructions and step-by-step decomposition structure, producing prompts that aim to explicitly guide the model through Context → Question → Reasoning → Answer sequences. BootstrapFewShot automatically curates high-quality demonstrations by bootstrapping examples from our training set, selecting demonstrations that maximize performance on our evaluation metric. This technique generates few-shot prompts containing up to four optimally chosen examples that are most representative of successful question-answering patterns in our domain. The optimization process operates on a subset of our 200-question synthetic dataset, using an 80/20 train-dev split with cross-validation. The resulting optimized prompt templates are serialized as JSON configurations and integrated into our generation pipeline, allowing dynamic selection between default prompting, DSPy-optimized CoT reasoning, and DSPy-optimized few-shot demonstration strategies. 3. Experimental Results 3.1. Retrieval Performance Analysis Table 1 presents our retrieval, document reformulation, and re-ranking results on our DataMorgana synthetic dataset.

Read the complete text