HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval DOI: 10.1145/1122445.1122456 Conference: Woodstock ’18: ACM Symposium on Neural Gaze Detection; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY Price: 15.00 ISBN: 978-1-45

Kevin Duh, Eugene Yang, Orion Weller, Andrew Yates, Dawn Lawrie · 2025

Read HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval DOI: 10.1145/1122445.1122456 Conference: Woodstock ’18: ACM Symposium on Neural Gaze Detection; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY Price: 15.00 ISBN: 978-1-45 by Kevin Duh, Eugene Yang, Orion Weller, Andrew Yates, Dawn Lawrie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22356.txt.

Introduction

Kevin Duh and co-authors describe their entry to a live question-answering contest in a 2025 paper. Their system had to research a question on the open web corpus, filter what it found, and answer with sources. A dense-token retriever searched a compressed index of a ten-billion-token web collection.

A mid-size open model drafted the search queries from context, and a small model filtered the hits. Up to nine passages fed the final answer writer. The entry placed fifth on automatic correctness scoring in the contest. The paper lays out the full pipeline as a recipe others can copy.

Catalog record

CREATORKevin Duh, Eugene Yang, Orion Weller, Andrew Yates, Dawn Lawrie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval DOI: 10.1145/1122445.1122456 Conference: Woodstock ’18: ACM Symposium on Neural Gaze Detection; June 03–05, 2018; Woodstock, NY Woodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 Kevin Duh, Eugene Yang, Orion Weller, Andrew Yates, Dawn Lawrie Abstract . The HLTCOE LiveRAG submission utilized the GPT-researcher framework for researching the context of the question, filtering the returned results, and generating the final answer. The retrieval system was a ColBERT bi-encoder architecture, which represents a passage with many dense tokens. Retrieval used a local, compressed index of the FineWeb10-BT collection created with PLAID-X, using a model fine-tuned for multilingual retrieval. Query generation from context was done with Qwen2.5-7B-Instruct, while filtering was accomplished with m2-bert-80M-8k-retrieval. Up to nine passages were used as context to generate an answer using Falcon3-10B. This system placed 5th in the LiveRAG automatic evaluation for correctness with a score of 1.07. Figure 1 . System design for the HLTCOE RAG system. 1. Introduction We adopted GPT-Researcher as the main framework for our LiveRAG submission to use an abstractive summarization paradigm to support answer generation. This approach was one of the best performing systems in TREC NeuCLIR’s report generation tasks ( Lawrie et al. 2025 ) . We used Qwen2.5-7B-Instruct 1 1 1 https://huggingface.co/Qwen/Qwen2.5-72B-Instruct ( Qwen et al. 2025 ) with 8 bit quantization for query formulation. We used a ColBERT-based search engine ( Nair et al. 2022 ) for retrieval with a PLAID-X model trained multilingually ( Yang et al. 2024 ) . The retrieval system returned top-ranked passages. Passages were then chunked into snippets. Passage snippets were filtered based on the cosine similarity between the original question and the snippet using m2-bert-80M-8k-retrieval 2 2 2 https://huggingface.co/togethercomputer/m2-bert-80M-8k-retrieval ( Fu et al. 2023 ) . The general architecture appears in Figure 1 . Finally, Falcon3-10B-Instruct 3 3 3 https://huggingface.co/tiiuae/Falcon3-10B-Instruct ( Team 2024 ) was used to generate the final response. The remainder of the paper outlines the detailed architecture of our system as well the prompts we used in the different steps in our pipeline. We describe the steps we took to validate the approach and optimize the performance to be able to perform at the required speed for the competition. 2. System Design Our RAG system divided the process into two main stages. In the first stage research was conducted to identify relevant information and in the second stage the response was written. Conducting research included generating queries based on an initial search and then gathering all the responsive data together from the retrieved passages. Writing the response used the gathered context to generate the final output of the system. In Section 2.1 we describe the process for conducting research. Section 2.2 describes the process of generating the response given the research. 2.1. Conduct Research The research stage involves four major steps: (1) issue an initial retrieval to set the context, (2) review the context to determine additional search queries, (3) issue additional search queries, and (4) divide passages into snippets and the snippets based on responsiveness to the original LiveRAG question. To set the initial context for the research, the top three ranked passages are used based on the query entered by the user, in this case the content of the question field for LiveRAG. Retrieval was done with an in-house index of the FineWeb-10BT collection. A ColBERT architecture search engine ( Nair et al. 2022 ) was used, which creates a dense vector per token for both the passage and query. We divided documents into non-overlapping passages of 450 tokens each for indexing. For efficiency, the index was compressed using the PLAID-X version 4 4 4 https://pypi.org/project/PLAID-X/ of ColBERT. Since it was possible for FineWeb-10BT to contain non-English documents, we chose to use an XLMR model 5 5 5 https://huggingface.co/FacebookAI/xlm-roberta-large ( Conneau et al. 2019 ) fine-tuned using translate distill ( Yang et al. 2024 ) for multilingual retrieval, which is trained with English queries and documents in several languages including English and Spanish. The training dataset is MS MARCO ( Nguyen et al. 2016 ) . MS MARCO documents were translated in-house with Sockeye Version 2 ( Domhan et al. 2020 ) to create the training data. The prompt in Figure 2 was used to determine additional queries that were used for retrieval. We set fields referenced in the prompt as follows: • max_iterations - 2 • task - the value of the question field • date - the date the software is run • context - a concatenation of the top three ranked passages using the question as the search query • dynamic_examples - based on max_iterations and in this case is "query 1", "query 2" This prompt was issued to the Qwen2.5-7B-Instruct-Turbo model. The top three passages were kept from each of the two queries generated as well as the original query. This created a list of nine passages. Our PLAID-X search engine was used to return passages for each the queries. The final step of conducting research utilized the m2-bert-80M-8k-retrieval model to embed both the original question and the passage in chunks of 1000 characters with 100 character overlap, referred to as snippets herein. The cosine similarity of the two vectors was computed. Any snippet whose similarity with the question was at least 0.35 was maintained for answer generation. Write {max_iterations} google search queries to search online that form an objective opinion from the following task: "{task}" Assume the current date is {date} if required. You are a seasoned research assistant tasked with generating search queries to find relevant information for the following task: "{task}". Context: {context} Use this context to inform and refine your search queries. The context provides real-time web information that can help you generate more specific and relevant queries. Consider any current events, recent developments, or specific details mentioned in the context that could enhance the search queries. You must respond with a list of strings in the following format: [ \{ dynamic_example \} ]. The response should contain ONLY the list. Figure 2 . Query Generation Prompt 2.2. Generate Response Information: "{context}" — Using the above information, answer the following query or task: "{question}" in one or two sentences. Use at most {total_words} words. Figure 3 . Response Generation Prompt Figure 4 . Histogram of number of unique documents (left), number of snippets (middle), and corresponding final prompt length (right) per input context for response generation In order to generate the response to the LiveRAG question, the snippets identified during the research phase are used as source material. In particular, the system role is left blank. The user role is the prompt in Figure 3 . This prompt has three attributes: • context - a concatenation of the text in each of the snippets that was accepted after the final filtering stage • question - the original LiveRAG question. • total_words - 200 A local version of Falcon3-10B-Instruct was used to generate the final response as specified in the rules for LiveRAG. The order of snippets in the prompt is not a rank order. Instead snippet ordering is determined by the rank order of the passage to the individual query that retrieved it as

Read the complete text