SIGIR 2025 – LiveRAG Challenge Report

David Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Oren Somekh, Ran TavoryTechnology Innovation Institute (TII), Haifa, Israel, Mehdi Ghissassi, Edo LibertyRoy Miara · 2025

Read SIGIR 2025 – LiveRAG Challenge Report by David Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Oren Somekh, Ran TavoryTechnology Innovation Institute (TII), Haifa, Israel, Mehdi Ghissassi, Edo LibertyRoy Miara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04942.txt.

Introduction

David Carmel and colleagues reported on the LiveRAG Challenge held at the SIGIR conference in 2025. Retrieval-augmented generation, where a language model answers questions using documents it looks up, improves fast, but comparing systems fairly is hard. The challenge fixed the ingredients: every team searched the same large web corpus and generated answers with the same open language model, so only retrieval and prompting strategies differed.

On the live challenge day, seventy teams from twenty-seven countries answered five hundred unseen questions within a strict two-hour window, supplying supporting evidence with each answer. Scoring ran in two stages, first an automated judge model rating correctness and faithfulness, then human review of the top submissions. Finalists were announced in June 2025 and prizes presented at the LiveRAG workshop in Padua, Italy.

Catalog record

CREATORDavid Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Oren Somekh, Ran TavoryTechnology Innovation Institute (TII), Haifa, Israel, Mehdi Ghissassi, Edo LibertyRoy Miara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SIGIR 2025 – LiveRAG Challenge Report David Carmel, Simone Filice, Guy Horowitz, Yoelle Maarek, Oren Somekh, Ran TavoryTechnology Innovation Institute (TII), Haifa, Israel, Mehdi Ghissassi, Edo LibertyRoy Miara Abstract The LiveRAG Challenge at SIGIR 2025, held between March and May 2025, provided a competitive platform for advancing Retrieval-Augmented Generation (RAG) technologies. Participants from academia and industry were invited to develop a RAG-based question-answering system using a fixed corpus (Fineweb-10BT) and a common open-source LLM (Falcon3-10B-Instruct). The goal was to facilitate challenging comparisons of retrieval and prompting strategies. During the Live Challenge Day, 70 teams from 27 different countries provided answers and supportive information to 500 unseen questions within a strict two-hour time window. Evaluation was conducted in two stages: first an automated LLM-as-a-judge approach was used to compute correctness and faithfulness score, then a manual review of top ranked submissions was conducted. The finalists were announced on June 12, 2025, with prizes awarded during the LiveRAG Workshop at SIGIR 2025 in Padua, Italy. 1 Overview Retrieval-Augmented Generation (RAG) has emerged as a widely accepted methodology for enhancing the effectiveness of large language models (LLMs), particularly for question-answering tasks ( lewis2020retrieval ; izacard2022few ; Gao+al:24a ) . Given a user request, a RAG system searches auxiliary sources to augment the request with relevant content ( lewis2020retrieval ) . RAG is attracting significant attention from the AI and IR communities, yet quality assessment of RAG systems is still an open challenge ( pradeep2025great ; es2024ragas ) . The goal of the LiveRAG Challenge 1 1 1 https://liverag.tii.ae/ was to allow research teams, across academia and industry, to advance their RAG research by evaluating their question answering solution and comparing the performance of their system with other teams, on a fixed external corpus (derived from the publicly available Fineweb 2 2 2 https://huggingface.co/datasets/HuggingFaceFW/fineweb ), and a fixed open-source LLM ( Falcon3-10B-Instruct 3 3 3 https://huggingface.co/tiiuae/Falcon3-10B-Instruct ), during a strict two-hour time window. Participants were expected to apply their own approach for key elements of the RAG system, such as query rewrite, text retrieval, prompt generation, etc., and integrate their solution with Falcon3-10B-Instruct for answer generation. Additionally, participant teams were given early access to TII’s DataMorgana ( filice2025generatingdiverseqabenchmarks ) , a synthetic Q&A generation tool, helping them generate benchmarks for training and testing their RAG systems. The shared corpus and the fixed generation model put the focus on the retrieval aspect of RAG, ensuring fair comparison of retrieval and prompting strategies. LiveRAG differs from similar competitions such as CRAG ( yang2024crag ) and the RAG Track in TREC 2024 ( thakur2025support ) in two key aspects. First, LiveRAG is ‘‘pseudo-live’’, hence its name, in order to mitigate the risk of over-tuning to the test set. Namely, participants had to submit their answers to a large set of unseen questions under a strict two-hour time limit. Second, in order to include participants who might not have access to expensive computational resources, selected teams were awarded $1500 Amazon AWS 4 4 4 https://aws.amazon.com/ compute credits, as well as $750 Pinecone 5 5 5 https://www.pinecone.io/ credits (graciously provided by our sponsors, AWS and Pinecone) for building their RAG solution. These resources, in addition to pre-built indices and complimentary access to the AI71 platform to run DataMorgana and Falcon3, significantly lowered the entry barrier to the Challenge, making LiveRAG both rigorous and inclusive. Following recent studies that highlight the limitation of LLM-as-a-judge approach to assess quality, rahmani2025 ; soboroff2025don we decided to apply a two-stage evaluation methodology. In first stage, the correctness and faithfulness scores (See § 4 ) of each submission were automatically computed via a carefully prompted strong LLM (different from the one used by the participants). Then manual assessment of more than half of the leading submissions was conducted by independent annotators. The finalists presented their results at the LiveRAG workshop at the SIGIR’2025 conference, during which winners were announced and prizes were awarded: $5000 for the winner, $3000 for the second, and $2000 for the third ranked system. 2 Challenge’s Resources In the following, we describe the main resources that were made available to participants during the challenge period. 2.1 External Content Source The Fineweb dataset ( penedo2024the ) consists of cleaned and de-duplicated Web content from CommonCrawl 6 6 6 https://commoncrawl.org . While Fineweb is relatively cleaner than other web-scale datasets, it still contains some toxic or offensive material and non-English pages, which increases the difficulty of the challenge. For the LiveRAG challenge, the RAG external repository was fixed to Fineweb-10BT 7 7 7 https://huggingface.co/datasets/HuggingFaceFW/fineweb/viewer/sample-10BT , a randomly sampled subset of 15M web documents from Fineweb. Although additional content sources were permitted, none of the teams chose to do so, as all evaluation questions were guaranteed to be answerable by Fineweb-10BT alone. 2.2 Pre-built retrieval indices Participants had the option to build their own search indices over Fineweb-10BT, or to use prebuilt sparse and dense indices. To prepare these indices, Fineweb-10BT documents were segmented into non-overlapping, sentence-based chunks of up to 512 tokens each, using the LlamaIndex sentence splitter 8 8 8 https://www.llamaindex.ai/ . After segmenting each document into sentences, consecutive sentences were aggregated, preserving sentence boundaries, until the chunk length limit was reached. • OpenSearch Sparse index: The resulting chunks were indexed into a BM25-based Sparse index implemented on the OpenSearch platform 9 9 9 https://opensearch.org , with OpenSearch default parameter settings. • Pinecone Dense index: Each chunk was embedded into a 768-dimensional vector using the E5-base-v2 model wang2024textembeddingsweaklysupervisedcontrastive . The embedding vectors were indexed using Pinecone’s Slab architecture ingber2025accurate . Under this architecture, every set of vectors is partitioned into non-overlapping, immutable components called “Slabs”. Data is first written to a Level-0 Slab, and compacted incrementally as new data is inserted. In practice, data is indexed in different ways depending on the Slab level and index size. Smaller corpora are indexed using a random projection algorithm ailon.2009 , while larger corpora are indexed using IVF-PQ jegou.2011 , and HNSW malkov.2020 . Although these latter algorithms are slower at indexing time, they provide significantly faster retrieval, especially for larger datasets. For the LiveRAG Challenge, this trade-off made them the preferred solution given the size of the corpus and the strict response time constraints. 2.3 Answer Generator One key design decision of the LiveRAG Challenge was to require all competitors to use the same LLM for answer generation, namely Falcon3-10B-Instruct, an instruction-tuned language model developed at the Technology Innovation Institute 10 10 10 https://www.tii.ae/ (TII), which, at the time of writing, achieves a state-of-the-art performance among LLMs of similar size. It represents a significant advancement in the Falcon family, trained on two Tera-tokens of diverse datasets and fine-tuned on 1.2 million samples of specialized content. Participating teams received free access to Falcon3-10B-Instruct through the AI71 platform. For other tasks in the RAG pipeline, besides answer generation, teams were allowed to use other LLMs, and computational tools in general, provided they do not exceed 10B parameters, to align with the challenge theme of using only moderately sizes resources for building a RAG system. 2.4 DataMorgana: Synthetic Benchmark Generator Participants were granted early access to DataMorgana ( filice2025generatingdiverseqabenchmarks ; filice2025dmacl ) , a novel tool that allows RAG developers to generate synthetic benchmarks from a

Read the complete text