Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities
2026
Read Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21937.txt.
Introduction
A benchmark described in this 2026 preprint asks a sharp question: can language models reason over scientific information that is genuinely new to them? Models score well on existing tests, but it is unclear how much of that is reasoning and how much is memory. Most evaluations score whole retrieval pipelines at once.
That design confounds reasoning with retrieval skill and tool choices, and scores are further polluted by memorized material and the shifting open web. The Retrieval-Infused Reasoning Sandbox instead separates retrieval from reasoning so each can be measured alone. Clean separation matters because a model that merely recalls cannot be trusted with the novel findings that real research produces every day.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities Abstract Despite strong performance on existing benchmarks, it remains unclear whether large language models can reason over genuinely novel scientific information. Most evaluations score end-to-end RAG pipelines, where reasoning is confounded with retrieval and toolchain choices, and the signal is further contaminated by parametric memorization and open-web volatility. We introduce DeR 2 , a controlled deep-research sandbox that isolates document-grounded reasoning while preserving core difficulties of deep search: multi-step synthesis, denoising, and evidence-based conclusion making. DeR 2 decouples evidence access from reasoning via four regimes—Instruction-only, Concepts (gold concepts without documents), Related-only (only relevant documents), and Full-set (relevant documents plus topically related distractors)—yielding interpretable regime gaps that operationalize retrieval loss vs. reasoning loss and enable fine-grained error attribution. To prevent parametric leakage, we apply a two-phase validation that requires parametric failure without evidence while ensuring oracle-concept solvability. To ensure reproducibility, each instance provides a frozen document library (drawn from 2023–2025 theoretical papers) with expert-annotated concepts and validated rationales. Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom: some models exhibit mode-switch fragility, performing worse with the Full-set than with Instruction-only, while others show structural concept misuse, correctly naming concepts but failing to execute them as procedures. † † affiliation: ByteDance Seed, M-A-P † † contribution: Full author list in Contributions † † correspondence: Ge Zhang at † † Project Page: https://retrieval-infused-reasoning-sandbox.github.io/ Figure 1 : Closed-book QA measures intrinsic knowledge, while standard RAG entangles retrieval and reasoning—distractors can hide whether errors come from bad retrieval or failed evidence-based reasoning. DeR 2 decouples the two by evaluating the same question under controlled inputs (instruction/concepts/related/full), isolating failure causes. 1 Introduction Recent foundation models have advanced from fluent assistants to strong problem solvers, achieving high performance on difficult, objectively graded tasks such as competition-style mathematics (e.g., AIME) and real-world software engineering (e.g., SWE-bench) 22 ; 14 . Meanwhile, agents—autonomous systems that navigate the internet to acquire and consolidate evidence—are increasingly used to extend models beyond parametric knowledge by coupling external search with internal reasoning and synthesizing information across multiple sources 20 ; 8 . Within this broader shift, deep search has emerged as a foundational evaluation scenario: it stresses an agent’s ability to uncover hard-to-find facts or entities through multi-step exploration, denoising, and evidence-driven synthesis. Motivated by these demands, recent work has proposed stronger agentic designs 18 ; 19 ; 28 ; 13 ; 17 and introduced more challenging benchmarks that better capture deep-research behavior, including BrowseComp for hard-to-find web browsing tasks and HLE for frontier, expert-level closed-ended questions 32 ; 24 . However, we find that current deep-research evaluation practice suffers from three major limitations. (1) End-to-end pipeline confounding with weak diagnosis: many benchmarks score the full stack—retrieval, reranking, summarization/compression, context stitching, and reasoning—so variance is often dominated by toolchain choices rather than the model’s latent research reasoning capability. Moreover, most setups expose only a binary “correct/incorrect” outcome, which makes it difficult to attribute errors to distinct failure types such as missing the key concept, identifying the concept but failing to apply it, or being diverted by topically related noise that shifts the reasoning starting point and triggers mode-switch derailment. As a result, these evaluations provide limited guidance for model selection, debugging, or targeted training. (2) Parametric leakage / memorization: if tasks can be solved via memorized facts, broad encyclopedic knowledge, or easily guessable numeric forms, then correctness does not reliably indicate that the model consulted the crucial evidence and reasoned with it, contaminating the evaluation signal. (3) Web volatility and irreproducibility: benchmarks that depend on open-web retrieval inherit non-stationarity across time, region, indexing updates, and page instability (movement, deletion, or content changes), producing high variance across repeated runs and hindering controlled comparisons and ablations 32 . To address these limitations, we propose DeR 2 , a controlled research sandbox that isolates document-grounded reasoning while preserving the core difficulties of deep search: multi-step synthesis, noise filtering, and evidence-based conclusion making. We mitigate (1) pipeline confounding and weak diagnosis by decoupling evidence access from reasoning via four evaluation regimes: Instruction-only (parametric knowledge), Concepts (instruction + gold concept set; concept composition/scheduling without document extraction), Related-only (instruction + only relevant documents; extraction and reasoning under clear evidence), and Full-set (relevant documents plus topically related distractors; denoising and reasoning). This design yields interpretable regime-to-regime gaps that operationalize “retrieval loss” versus “reasoning loss,” and it supports error attribution, including missing concepts, concept misuse, and noise-induced mode switching. We mitigate (2) parametric leakage through a rigorous two-phase validation protocol: each problem must exhibit parametric failure (low accuracy without documents) while remaining document-solvable (a correct solution is achievable when provided oracle concepts). This jointly enforces novelty (not answerable from memorized knowledge alone) and tractability (answerable given the intended evidence). Finally, we address (3) web volatility by replacing open-web dependence with a frozen, per-instance document library (mean: 6.5 documents) drawn from 2023–2025 theoretical papers. Each library includes both necessary evidence and carefully selected distractors that mimic realistic, imperfect literature gathering. Each instance further provides expert-annotated concept sets and validated chain-of-thought rationales, enabling process-level evaluation of evidence usage rather than relying solely on final-answer accuracy. Our experiments reveal two consistent failure patterns. (1) Mode-switch fragility: Some models perform better under Instruction-only than under Full-set, indicating that additional documents can disrupt the controller responsible for shifting from parametric reasoning to evidence-grounded reasoning, thereby degrading performance. (2) Structural concept misuse: even when models identify and restate the correct concepts, they often fail to execute them as procedures (e.g., applying rules step-by-step or instantiating abstractions), instead reverting to generic heuristics that appear plausible but produce incorrect conclusions. Our contributions are threefold: • We articulate desiderata for evaluating research-oriented LLMs and show that prior QA, RAG, and reasoning benchmarks do not sufficiently isolate evidence-grounded synthesis. • We release DeR 2 , a rigorously curated benchmark featuring frozen document libraries with embedded noise, expert concept annotations, and validated CoT rationales drawn from frontier scientific literature. • We provide a systematic analysis of leading LLMs under this controlled setup, revealing substantial headroom even for top-tier systems and identifying concrete failure modes that future architectures and training regimes must address. 2 Benchmark 2.1 Benchmark Overview We introduce DeR 2 , a controlled benchmark for evaluating retrieval-infused reasoning in scientific problem solving. The benchmark is designed to expose failure modes that are difficult to diagnose in end-to-end retrieval-augmented generation (RAG) or agent-based deep research evaluations. Figure 2 : (a) Existing benchmarks. Closed-book QA only evaluates intrinsic (parametric) knowledge. RAG pipelines couple retrieval and reasoning end-to-end, creating a reasoning–retrieval tangle that masks the true cause of failure. (b) Decoupled Retrieval and Reasoning Benchmark. DeR 2 provides a controlled setting that decouples retrieval from reasoning by comparing Instruction-only, Full DocSet (related + noise), Related-only, and Concepts-only conditions, enabling diagnosis of grounding, mode-switching, and concept execution errors. DeR 2 explicitly separates (a) a model’s ability to identify and extract solution-relevant concepts from a document set, from (b) its ability to compose and schedule these concepts to derive the correct answer. This decomposition enables controlled diagnosis of where performance collapses in retrieval-infused reasoning. Compared to conventional RAG benchmarks, DeR 2 provides (i) a controlled document set with both