Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, Australia. Proceedings of the 49th Int’l ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, Australia

Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani · 2026

Read Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, Australia. Proceedings of the 49th Int’l ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, Australia by Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18516.txt.

Introduction

Rafiee and co-authors introduce Total Recall QA, a verifiable test suite for deep research agents. Deep research systems seek and reason over many open sources, yet their evaluation lacks set standards. They list requirements for sound evaluation and find past benchmarks fall short. TRQA builds single-answer, total recall queries from a structured knowledge base paired with a text corpus. Wikidata-Wikipedia and a synthetic e-commerce source supply the data, with baseline results set.

Catalog record

CREATORMahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39788
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18516

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, Australia. Proceedings of the 49th Int’l ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, Australia Mahta Rafiee, Heydar Soudani, Zahra Abbasiantaeb, Mohammad Aliannejadi, Faegheh Hasibi, Hamed Zamani Abstract . Deep research agents have emerged as LLM-based systems designed to perform multi-step information seeking and reasoning over large, open-domain sources to answer complex questions by synthesizing information from multiple information sources. Given the complexity of the task and despite various recent efforts, evaluation of deep research agents remains fundamentally challenging. This paper identifies a list of requirements and optional properties for evaluating deep research agents. We observe that existing benchmarks do not satisfy all identified requirements. Inspired by prior research on TREC Total Recall Tracks, we introduce the task of Total Recall Question Answering and develop a framework for deep research agents evaluation that satisfies the identified criteria. Our framework constructs single-answer, total recall queries with precise evaluation and relevance judgments derived from a structured knowledge base paired with a text corpus, enabling large-scale data construction. Using this framework, we build TRQA, a deep research benchmark constructed from Wikidata-Wikipedia as a real-world source and a synthetically generated e-commerce knowledge base and corpus to mitigate the effects of data contamination. We benchmark the collection with representative retriever and deep research models and establish baseline retrieval and end-to-end results for future comparative evaluation. *[inlinelist,1]label=(), 1. Introduction Figure 1 . An example from the TRQA benchmark. Recent advances in large language models (LLMs) have enabled a new class of agentic systems that integrate reasoning, tool use, and external information access to tackle complex, real-world tasks. Among these, Deep Research Agents (DRAs) have emerged for performing multi-step information seeking, iterative search planning, and reasoning over large, open-domain corpora in order to answer complex questions ( 20 ; 17 ) . However, robust evaluation of deep research agents remains fundamentally challenging. To systematically assess the quality of existing evaluation methodologies and benchmarks, we identify a taxonomy of requirements and optional criteria. In more detail, we argue that a reliable and reusable methodology for evaluating DRAs should satisfy the following requirements: (1) Multi-source : The answer to a deep research question is a synthesis of information from multiple, and sometimes many, sources (e.g., documents), involving complex reasoning and information retrieval processes. (2) Verifiability : Metrics for evaluating the generated answers must be transparent, reliable, and verifiable. (3) Reproducibility : Evaluation must be reproducible. Besides these requirements, the following optional criteria would substantially enhance the robustness and value of a deep research evaluation methodology: (4) Retrieval Evaluation: The evaluation methodology should provide a reliable, reproducible, and verifiable evaluation of intermediate retrieval steps . (5) Generalizability Evaluation : The evaluation methodology should measure generalizability of DRAs across diverse aspects, including data contamination, memorization of popular information, and cross-domain generalization. Given the needs of the tech industry and the technical challenges introduced by this task, researchers have recently introduced multiple datasets for evaluating Deep Research Agents. However, they do not meet all the necessary requirements mentioned above. We refer the reader to Table 1 for a list of existing popular benchmarks for Deep Research Agents and their characteristics. For instance, many benchmarks, e.g., ( 8 ; 5 ; 21 ) , evaluate deep research agents through long-form reports or citation-rich answers. In these settings, the evaluation is usually done using rubric-based and LLM-based scoring ( 31 ; 8 ; 5 ; 21 ; 19 ) , which often do not satisfy the Verifiability and/or Reproducibility requirements mentioned above. Furthermore, they are often task-specific and do not generalize, thus conflicting with the Generalizability Evaluation criterion. Furthermore, source verification of citations is typically limited to a subset of claims with explicit citation markers ( 38 ; 8 ; 21 ) , and therefore fails to effectively assess factual correctness across the full report ( 14 ) . Other than the difficulty in evaluation of the generated answer, end-to-end evaluation of many deep research systems is done using live black-box web search APIs, which again conflicts with the Verifiability and Reproducibility requirements. The risk of data contamination and LLM memorization ( 10 ; 7 ) is another challenge in creating effective test collections which is often overlooked in many of the existing benchmarks ( 5 ; 39 ) . Figure 2 . An overview of the TRQA data generation framework for Total Recall QA tasks. Table 1 . Comparison of existing deep research benchmarks and TRQA across key requirements and criteria. Benchmark Multi-source Verifiable Reproducible Retrieval Generalizable BrowseComp ( 39 ) ✓ ✓ ✗ ✗ ✗ BrowseComp-Plus ( 4 ) ✓ ✓ ✓ ✓ ✗ DeepResearchBench ( 8 ) ✓ ✗ ✗ ✗ ✓ DeepResearchGym ( 5 ) ✓ ✗ ✗ ✗ ✗ ReportBench ( 21 ) ✓ ✗ ✗ ✗ ✗ ScholarGym ( 32 ) ✓ ✓ ✓ ✓ ✗ DeepSearchQA ( 13 ) ✓ ✓ ✗ ✓ ✗ TRQA ✓ ✓ ✓ ✓ ✓ These limitations highlight the need for evaluation frameworks that support reproducible component-level assessment of deep research systems, precise and reliable scoring methodologies, explicit relevance judgments, and fixed corpora that enable both end-to-end and intermediate retrieval evaluation. To address this need and inspired by the TREC Total Recall Tracks ( 30 ; 11 ) , we introduce the task of Total Recall Question Answering (QA) , which enables building a verifiable and reproducible framework for DRAs: Total Recall QA is a question answering task where accurate generation of the answer requires retrieving all relevant documents for a given question from a large corpus, as well as reasoning and synthesizing information across all relevant documents. See Figure 1 for an illustration of a Total Recall QA example. This paper introduces an evaluation suite for the Total Recall QA task, consisting of: (1) A framework for systematic generation of datasets for the task; we propose an entity-centric framework that leverages a paired structured knowledge base and text corpus to generate total recall queries and explicit relevance judgments. (2) Three datasets constructed using the proposed framework; we focus on single-response questions with numerical answers to satisfy the Verifiability requirement. While offering simple evaluation metrics, our datasets preserve the core difficulty of deep research tasks by requiring total recall retrieval and multi-step reasoning for resolving the queries. To mitigate the effects of data contamination, our datasets consist of a synthetically generated e-commerce dataset in addition to two datasets based on Wikipedia articles. An example query from our datasets is presented in Figure 1 . (3) Benchmark results and extensive analysis of intermediate retrieval and end-to-end performance of state-of-the-art systems; we conduct empirical evaluation of a diverse set of commercial and open-source state-of-the-art LLMs and deep research agents, showing that competitive deep research methods and LLMs struggle to answer these questions, even with oracle retrieval results. Failure analysis of models with oracle retrieval suggests that most failures are due to reasoning errors rather than parametric knowledge bias. Results suggest that TRQA contains challenging datasets for both evaluation of retrieval and reasoning aspects of DRAs.

Read the complete text