A Reproducibility Study of Metacognitive Retrieval-Augmented Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808551 ISBN: 979-8-4007-2599-9/2026/07 CCS: Computing methodologies Natural language generation
Gabriel Iturra Bocaz, Petra Galuščáková · 2026
Read A Reproducibility Study of Metacognitive Retrieval-Augmented Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808551 ISBN: 979-8-4007-2599-9/2026/07 CCS: Computing methodologies Natural language generation by Gabriel Iturra Bocaz, Petra Galuščáková (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19899.txt.
Introduction
Iturra-Bocaz and Galuscakova re-ran MetaRAG, a retrieval framework in which a language model critiques its own reasoning to judge when to stop searching. Their rerun confirms its edge over standard and reasoning-based baselines, but raw scores fall short of the first report. They blame closed-model updates, missing details, and unreleased prompts. Added rerankers lift scores a lot, and MetaRAG holds up better than the SIM-RAG critic method when more retrieval features join.
Catalog record
| CREATOR | Gabriel Iturra Bocaz, Petra Galuščáková |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47319 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Reproducibility Study of Metacognitive Retrieval-Augmented Generation Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808551 ISBN: 979-8-4007-2599-9/2026/07 CCS: Computing methodologies Natural language generation Gabriel Iturra Bocaz, Petra Galuščáková Abstract . Recently, Retrieval Augmented Generation (RAG) has shifted focus to multi-retrieval approaches to tackle complex tasks such as multi-hop question answering. However, these systems struggle to decide when to stop searching once enough information has been gathered. To address this, 52 introduced Metacognitive Retrieval Augmented Generation (MetaRAG), a framework inspired by metacognition that enables Large Language Models to critique and refine their reasoning. In this reproducibility paper, we reproduce MetaRAG following its original experimental setup and extend it in two directions: (i) by evaluating the effect of PointWise and ListWise rerankers, and (ii) by comparing with SIM-RAG, which employs a lightweight critic model to stop retrieval. Our results confirm MetaRAG’s relative improvements over standard RAG and reasoning-based baselines, but also reveal lower absolute scores than reported, reflecting challenges with closed-source LLM updates, missing implementation details, and unreleased prompts. We show that MetaRAG is partially reproduced, gains substantially from reranking, and is more robust than SIM-RAG when extended with additional retrieval features. † † cc-license: by 1. Introduction Metacognition is the ability to reflect on and critique one’s own cognitive processes ( 19 ) . It enables individuals to notice gaps in understanding, judge whether additional information is needed, and adjust their strategies to reach more reliable conclusions. This self-awareness is fundamental in tasks that demand several reasoning steps across multiple pieces of information, allowing errors to be detected and planning strategies to be refined ( 34 ; 19 ) . Such ability is important because it helps to maintain reasoning that stays focused on the question rather than drifting to irrelevant details, and that remains consistent when integrating new or conflicting information ( 7 ) . However, although modern LLMs employ implicit control mechanisms such as routing or model selection, they typically generate responses without explicit self-reflection or awareness of their own knowledge limitations ( 23 ; 37 ) . Inspired by the concept of metacognition, 52 proposed MetaRAG, a framework that enables RAG systems to monitor and critique their own reasoning before further retrieval is required. While other reasoning-oriented approaches such as CoT ( 44 ) , ReAct ( 47 ) , or Self-Ask ( 30 ) focus on guiding the reasoning process, they do not explicitly implement metacognitive control. This distinction remains important because multi-stage retrieval systems must regulate additional retrieval to avoid unnecessary overhead, latency, and cost, especially in large-scale and industrial settings. In this reproducibility paper, we reproduce MetaRAG’s results under a comparable setup and further investigate how aspects such as document ordering relate to metacognitive reasoning in LLMs through reranker models. In this work, we formulate and address the following research questions: - RQ1 : How reproducible e are the results of MetaRAG? While the original MetaRAG paper reports strong results on multi-hop Question-Answering (QA) tasks ( 25 ) , its reproducibility remains uncertain ( 20 ) . For example, the code and prompts for the baseline implementations are not fully available, and the underlying LLMs have evolved since the study was published 1 1 1 https://platform.openai.com/docs/deprecations#2023-11-06-chat-model-updates . Moreover, some aspects of the original setup remain unclear, such as how sparse and dense retrieval are combined in the implementation and described in the paper ( 52 ) . Because MetaRAG represents an important step toward integrating metacognitive control into RAG systems, we aim to re-examine how it performs under current experimental conditions, including updated LLM backends, retrieval implementations, and incomplete or underspecified setup details. - RQ2 : Can reranker models improve the performance of MetaRAG? In the human reasoning process for answering complex questions, we first prioritize information and then reflect on it ( 27 ) . Similarly, while MetaRAG critiques its own reasoning, it does not explicitly address the quality or ordering of retrieved documents. Since reranker models help reduce noise and control order ( 6 ; 49 ; 26 ) , we investigate whether integrating them into MetaRAG enhances its overall reasoning and answer quality. - RQ3 : How does MetaRAG compare to other metacognitive RAG frameworks, such as SIM-RAG? While MetaRAG investigates metacognition in multi-hop QA, the comparative impact of different metacognitive mechanisms on retrieval and reasoning is still not well understood. For example, 45 argue that multi-round retrieval RAG systems do not know when they have gathered enough information to answer a question. They propose SIM-RAG, a framework that adds a lightweight critic module which continuously checks if enough information has been retrieved, helping the system decide when to stop searching and start reasoning. We therefore compare MetaRAG with SIM-RAG to better understand how different forms of metacognition control influence retrieval stopping and reasoning performance. Main Contributions. Our main contributions are summarized as follows: (1) we conduct a reproducibility study of the MetaRAG framework, focusing on evaluating its performance on QA tasks and comparing it with other baselines, (2) we extend MetaRAG with rerankers, finding performance gains by reducing noise and mitigating the effects of document reordering, and (3) we compare MetaRAG with SIM-RAG and show that MetaRAG, especially when combined with rerankers, performs more robustly. To support further research, we release our code 2 2 2 https://github.com/iai-group/sigir2026-metarag and experimental setup, including baselines. 2. Related Work RAG has become a central paradigm for enhancing LLMs by injecting external information, which extends their internal knowledge and addresses common issues such as hallucinations and unverifiable answers ( 21 ; 10 ) . Early RAG systems employed single-retrieval strategies ( 8 ; 10 ; 21 ; 51 ) to answer simple queries, such as factual questions, but these approaches fall short for multi-hop QA tasks that require several reasoning steps to produce high-quality answers. To overcome these challenges, research has shifted towards multi-retrieval frameworks ( 17 ; 40 ; 18 ; 47 ) , in which the retrieval stage is invoked iteratively during reasoning or generation. Generally, these approaches can be broadly divided into passive retrieval, decompositional calls, and dynamic schemes controlled by LLMs. Passive Retrieval . In these approaches, retrieval is triggered on a fixed schedule (e.g., after every sentence or token count), regardless of whether the system needs to retrieve information at that moment. For example, 17 enhance traditional language models by consulting a datastore of past contexts to recall similar contexts for informing the next predictions. Similarly, 40 introduced IR-CoT, which performs retrieval after every Chain of Thought (CoT) reasoning step. Decompositional Queries . In these systems, complex queries are divided into sub-questions, each triggering a dedicated retrieval process, the results of which are later aggregated. 18 proposed that rather than solving a complex problem all at once, the original question should be decomposed into a sequence of simpler sub-queries, with each sub-question performing retrieval independently. Retrieval controlled by LLMs . In these frameworks, retrieval is guided by the reasoning steps of LLMs. For example, ReAct ( 47 ) combines CoT reasoning to