IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction
Guoxin Chen, Zile Qiao, Xuanzhong Chen, Donglei Yu, Haotian Xu, Wayne Xin Zhao, Ruihua Song, Wenbiao Yin, Huifeng Yin, Liwen Zhang, Kuan Li, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou · 2025
Read IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction by Guoxin Chen, Zile Qiao, Xuanzhong Chen, Donglei Yu, Haotian Xu, Wayne Xin Zhao, Ruihua Song, Wenbiao Yin, Huifeng Yin, Liwen Zhang, Kuan Li, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07327.txt.
Introduction
Guoxin Chen and fifteen co-authors rethink how deep-research agents work on long tasks. Current agents stuff everything into one ever-growing context window until context suffocation and noise contamination choke the reasoning. IterResearch replaces accumulation with iteration: an evolving report serves as memory. The workspace is periodically rebuilt by synthesizing what has been learned so far, keeping reasoning capacity constant across arbitrary exploration depths.
Training uses an efficiency-aware policy optimization that discounts rewards geometrically to reward efficient exploration, with adaptive downsampling for stable distributed training. The agent improved over open-source rivals by about fourteen and a half points on average across six benchmarks and kept scaling to over two thousand interactions in testing. Used purely as a prompting strategy, it also lifted frontier models greatly over ReAct on long-horizon tasks.
Catalog record
| CREATOR | Guoxin Chen, Zile Qiao, Xuanzhong Chen, Donglei Yu, Haotian Xu, Wayne Xin Zhao, Ruihua Song, Wenbiao Yin, Huifeng Yin, Liwen Zhang, Kuan Li, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07327
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction Guoxin Chen, Zile Qiao, Xuanzhong Chen, Donglei Yu, Haotian Xu, Wayne Xin Zhao, Ruihua Song, Wenbiao Yin, Huifeng Yin, Liwen Zhang, Kuan Li, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou Abstract Recent advances in deep-research agents have shown promise for autonomous knowledge construction through dynamic reasoning over external sources. However, existing approaches rely on a mono-contextual paradigm that accumulates all information in a single, expanding context window, leading to context suffocation and noise contamination that limit their effectiveness on long-horizon tasks. We introduce IterResearch, a novel iterative deep-research paradigm that reformulates long-horizon research as a Markov Decision Process with strategic workspace reconstruction. By maintaining an evolving report as memory and periodically synthesizing insights, our approach preserves consistent reasoning capacity across arbitrary exploration depths. We further develop Efficiency-Aware Policy Optimization (EAPO), a reinforcement learning framework that incentivizes efficient exploration through geometric reward discounting and enables stable distributed training via adaptive downsampling. Extensive experiments demonstrate that IterResearch achieves substantial improvements over existing open-source agents with average +14.5pp across six benchmarks and narrows the gap with frontier proprietary systems. Remarkably, our paradigm exhibits unprecedented interaction scaling, extending to 2048 interactions with dramatic performance gains (from 3.5% to 42.5%), and serves as an effective prompting strategy, improving frontier models by up to 19.2pp over ReAct on long-horizon tasks. These findings position IterResearch as a versatile solution for long-horizon reasoning, effective both as a trained agent and as a prompting paradigm for frontier models. † † footnotetext: ∗ Corresponding Authors. Code Figure 1: Performance of IterResearch against state-of-the-art open-source long-horizon agents. 1 Introduction Recent advances in deep-research agents represent a transformative shift for Large Language Models (LLMs), moving beyond passive knowledge acquisition from the model itself towards autonomous agents that construct knowledge through dynamic reasoning over external sources [ 22 , 9 , 33 , 24 , 1 , 20 ] . These frontier proprietary systems have demonstrated remarkable performance on long-horizon tasks that require sustained reasoning and information-seeking capabilities over extended interactions. When tackling long-horizon tasks, recent works [ 4 , 28 , 45 , 13 , 16 , 14 , 29 ] typically append all retrieved information and intermediate reasoning steps to a single, continuously expanding context window, which we term the mono-contextual paradigm . While straightforward to implement, this paradigm fundamentally undermines the sustained reasoning capabilities required for long-horizon tasks: (1) context suffocation : as the context window fills with all prior interactions, the available space for model reasoning progressively shrinks, forcing increasingly constrained responses that ultimately degrade into premature or superficial conclusions. (2) noise contamination : irrelevant information from web searches and early exploration errors become permanently embedded in the context, creating cascading interference that dilutes signal quality throughout the entire reasoning process. To address these limitations, we introduce IterResearch , a novel Iterative Deep-Research Paradigm that fundamentally reimagines how autonomous agents maintain sustained reasoning capacity in long-horizon scenarios. Our key insight is that effective long-horizon research requires periodic synthesis and strategic forgetting—capabilities absent in current mono-contextual approaches. Specifically, we extend the Markov Decision Process (MDP) framework for deep research with a distinctive state design: rather than maintaining an ever-expanding history, each state is a strategically reconstructed workspace containing only essential elements: the question, an evolving report serving as the agent’s memory, and the immediate context needed for current reasoning. This Markovian structure, where future exploration depends only on the current reconstructed state rather than the entire history, enables the agent to maintain consistent reasoning capacity across arbitrary exploration depths while naturally circumventing the degradation that plagues mono-contextual approaches. To fully realize this paradigm’s potential, we develop Efficiency-Aware Policy Optimization (EAPO) , a reinforcement learning framework specifically designed for training IterResearch. EAPO addresses two critical challenges unique to our iterative paradigm: First, recognizing that not all successful trajectories are equally valuable, we introduce efficiency-aware rewards that geometrically discount based on trajectory length—agents reaching correct conclusions through concise, focused exploration receive higher rewards than those requiring extensive iterations. Second, since our paradigm naturally decomposes trajectories into independent training samples at each round, we employ adaptive downsampling to handle the variable sample counts based on data-parallel size, ensuring stable distributed training while preserving over 99% of training data. Extensive experiments demonstrate that IterResearch significantly outperforms existing open-source agents, achieving an average improvement of 14.5 percentage points (pp) across six challenging benchmarks. More remarkably, IterResearch narrows the performance gap with frontier proprietary systems, even surpassing some on these benchmarks. Furthermore, our work reveals three fundamental insights about deep-research agents. First, our iterative paradigm unlocks extreme interaction scaling —a capability theoretically extensible to infinite depths yet structurally infeasible for current mono-contextual approaches. To our knowledge, we are the first to successfully extend agents to 2048 interactions with only 40K context length, exhibiting dramatic performance improvements (3.5% → 42.5%) as maximum interactions increase from 2 to 2048, suggesting that the perceived difficulty of long-horizon tasks may stem from insufficient exploration capacity. Second, we observe cross-paradigm knowledge transfer : trajectories generated by IterResearch significantly enhance mono-contextual agents, demonstrating that our paradigm induces superior exploration behaviors that create high-quality training signals transferable even across paradigmatically different approaches. Third, our iterative paradigm serves as an effective prompting strategy : without any training, simply applying it to frontier models yields substantial improvements over the standard mono-contextual approach, ReAct [ 38 ] , particularly on long-horizon tasks (+12.7-19.2pp on BrowseComp), revealing that IterResearch offers a model-agnostic solution to long-horizon reasoning. These results confirm the effectiveness of our iterative paradigm in enabling both deeper exploration and higher-quality reasoning in long-horizon scenarios. In summary, our main contributions can be summarized as follows: • We propose IterResearch, a novel iterative deep-research paradigm that reformulates long-horizon research as an MDP with strategic workspace reconstruction, maintaining sustained reasoning capacity through periodic synthesis and an evolving report memory—eliminating the context suffocation and noise contamination that plague mono-contextual approaches. • We develop Efficiency-Aware Policy Optimization (EAPO) with geometric discounted rewards that incentivize efficient exploration and adaptive downsampling for stable distributed training, enabling effective learning from our paradigm’s unique trajectory structure. • We demonstrate IterResearch’s exceptional capabilities and broader impact: (1) achieving an average 14.5 pp improvement across six challenging benchmarks; (2) exhibiting interaction scaling to 2048 interactions with dramatic performance gains; (3) enabling cross-paradigm knowledge transfer to enhance mono-contextual agents; (4) providing a model-agnostic prompting strategy that significantly improves frontier models on long-horizon tasks without training. 2 Related Work Retrieval-Augmented Generation (RAG). RAG is a crucial approach to overcome knowledge limitations of large language models (LLMs) by integrating external information sources [ 21 , 40 , 2 , 31 , 4 , 13 , 28 , 45 ] . However, traditional RAG methods are typically confined to static retrieval environments, such as Wikipedia, with limited exploration spaces, making them inadequate for complex, long-horizon reasoning tasks that require dynamic information gathering. Deep Research. Recent advances in deep research [ 22 , 9 , 33 , 20 ] have transcended RAG’s limitations by deploying autonomous agents in real-world environments, demonstrating remarkable capabilities in navigating complex web environments and synthesizing information from diverse sources. However, existing open-source methods [ 15 , 16 , 29 , 14 ] predominantly adopt a