WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning
Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang · 2026
Read WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning by Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03164.txt.
Introduction
Xinmiao Yu and six co-authors stabilize the planning of web agents, language model systems that search the web over many steps. Their analysis uncovers a plan anchor effect: the first reasoning step shapes everything downstream far more than later steps, yet standard reinforcement learning spreads rewards evenly across the whole trajectory. Their framework, Anchor-GRPO, therefore splits training into two stages.
In the first stage, the agent optimizes its opening plan against detailed rubrics drawn from self-play and human calibration. In the second, execution is trained with sparse rewards to stay aligned with that plan and use tools efficiently. Across four benchmarks and models from three to thirty billion parameters, Anchor-GRPO beat its baselines on task success and tool efficiency, with the largest model reaching strong scores on BrowseComp and GAIA.
Catalog record
| CREATOR | Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03164
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning Thanks: Correspondence. Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang Abstract Large Language Model(LLM)-based agents have shown strong capabilities in web information seeking, with reinforcement learning (RL) becoming a key optimization paradigm. However, planning remains a bottleneck, as existing methods struggle with long-horizon strategies. Our analysis reveals a critical phenomenon— plan anchor —where the first reasoning step disproportionately impacts downstream behavior in long-horizon web reasoning tasks. Current RL algorithms, fail to account for this by uniformly distributing rewards across the trajectory. To address this, we propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution. In Stage 1, the agent optimizes its first-step planning using fine-grained rubrics derived from self-play experiences and human calibration. In Stage 2, execution is aligned with the initial plan through sparse rewards, ensuring stable and efficient tool usage. We evaluate Anchor-GRPO on four benchmarks: BrowseComp, BrowseComp-Zh, GAIA, and XBench-DeepSearch. Across models from 3B to 30B, Anchor-GRPO outperforms baseline GRPO and First-step GRPO, improving task success and tool efficiency. Notably, WebAnchor-30B achieves 46.0% pass@1 on BrowseComp and 76.4% on GAIA. Anchor-GRPO also demonstrates strong scalability, getting higher accuracy as model size and context length increase. 1 Introduction Reinforcement Learning (RL) have significantly enhanced the capabilities of autonomous, tool - augmented agents built on large language models (LLMs) for web information seeking ( 12 ; 13 ; 31 ) . These agents often referred to as deep research ( 18 ; 10 ; 24 ) , go beyond static retrieval and instead iteratively formulate queries, invoke external tools, and collect evidence from diverse online sources to answer complex questions. (a) The plan anchor phenomenon, where the first-step decision disproportionately affects the trajectory’s success. (b) Impact of the first step on downstream task accuracy. (c) Plan rubrics visualized through a radar chart. Figure 1: Illustrating the plan anchor phenomenon, the critical role of the first step in task accuracy, and the use of plan rubrics to guide optimization. Despite recent advancements, deep research agents still struggle with long-horizon planning and maintaining strategy coherence ( 7 ; 19 ) . Increasing tool usage or managing context alone does not ensure consistent multi-step reasoning; instead, accumulated errors often lead to performance degradation ( 16 ) . While existing reinforcement learning (RL) methods—such as context management, reward shaping, and entropy-based optimization—mitigate certain aspects, there are still gaps to address planning instability and the lack of a comprehensive long-term strategy ( 3 ; 27 ; 33 ; 4 ) . A crucial insight is that the ability to maintain coherent long-horizon behavior hinges on early decisions ( 22 ) . Recent work highlights that not all reasoning steps are equally important ( 1 ) . In particular, the first planning step often acts as a structural anchor—shaping exploration, tool usage, and evidence integration ( 5 ) . Initial missteps can trigger cascading failures and destabilize the entire trajectory ( 23 ; 20 ) . Motivated by this, we identify a critical phenomenon in long-horizon web agents, which we term the plan anchor , as shown in Fig. 1(a) . Our motivation experiments (Fig. 1(b) ) demonstrate substantial performance drops in Avg Pass@1 across BC-ZH, BC-EN, and GAIA benchmarks due to incorrect first steps—by 28.7%, 30.9%, and 23.6%, respectively. These results underscore the importance of designing reward schemes that explicitly prioritize the impact of the first step. To address this, we introduce Anchor-GRPO , a two-stage RL framework that separates planning and execution. The first stage focuses on optimizing the initial planning step. In this stage, we develop the Plan Rubrics Learner , by analyzing both successful and failed experiences, the learner adaptively refines essential planning criteria, such as task decomposition, goal alignment, and tool selection. The final rubrics assign higher scores to correct plans, as shown in Figure 1(c) , confirming the learner’s ability to capture important planning capabilities. These refined rubrics are incorporated as reward signals in the RL process, guiding the agent to improve its planning capabilities. In Stage 2, the focus shifts to execution. Sparse rewards align execution with the initial plan, ensuring stability throughout the reasoning process and addressing long-horizon credit assignment challenges. By combining the power of plan rubrics and execution alignment, Anchor-GRPO enables agents to plan first, then act , resulting in a more reliable approach which can maintain consistency over long reasoning trajectories. Our contributions are threefold: • Plan Anchor Phenomenon in Long-Horizon Web Reasoning: We identify the critical phenomenon of plan anchor , where a first-step decision disproportionately impacts the success of the entire trajectory, highlighting the importance of the initial planning step in long-horizon web reasoning. • Experience-based Plan Rubrics Learner: We propose the Plan Rubrics Learner framework, which adaptively learns key dimensions of effective long-horizon plans from self-play experiences. The learnt rubrics will guide the agent in optimizing the first planning step for improved reasoning and task success. • Anchor-GRPO and Its Superior Performance: We introduce Anchor-GRPO, a two-stage RL framework that enhances task success and long-horizon reasoning, outperforming existing methods on several challenging benchmarks. It also demonstrates scalability across model sizes and context lengths, making it suitable for more complex tasks. 2 Preliminary Agentic Web Reasoning. We follow the standard ReAct-style agentic workflow ( 30 ) , where an LLM-based agent interleaves Thought , Action , and Observation . At step t , the agent reads the trajectory history \mathcal{H}_{t-1} and produces a reasoning trace \tau_{t} and an executable action a_{t} , after which the environment returns an observation o_{t} . A T -step rollout is defined as: \mathcal{H}_{T}=(q,\tau_{1},a_{1},o_{1},\dots,\tau_{T},a_{T},o_{T},\tau_{T+1},a_{T+1}), where q is the task query and a_{T+1} denotes the final answer. At each step, the policy model samples: (\tau_{t},a_{t})\sim\pi_{\theta}(\cdot\mid\mathcal{H}_{t-1}). Tool Design. Following standard web agent designs ( 13 ; 8 ) , we define the action space with two tools for web exploration: • Search : This tool issues top- k web search queries to retrieve relevant snippets and URLs. It accepts natural language inputs and returns structured results from the Google Search API, including titles, snippets, and hyperlinks. • Visit : Given a specific URL, the agent can browse the full page content and extract factual evidence. We use a language model browser to simulate document-level reading and structured content extraction. Each action a_{t} is thus instantiated as either Search(query) or Visit(url) . These tools enable multi-hop evidence gathering and decision-making under long-horizon uncertainty. 3 Methodology In this section, we introduce a two-stage training framework Anchor-GRPO to optimize long-horizon reasoning in web agents. In the first stage, a Plan Rubrics Learner is used to derive effective planning criteria from past experiences, which are used to optimize the agent’s initial planning with a dense reward. In the second stage, the agent’s execution is optimized based on sparse rewards, with both stages jointly trained to align planning and execution. 3.1 Anchor-GRPO Overview Anchor-GRPO is a two-stage reinforcement learning framework that integrates planning and execution within a single policy model. In the first stage, the model acts as a Planner ( \pi_{\text{planner}} ), generating an initial plan from the user query, optimized with dense rewards to improve task decomposition and planning quality. In the second stage, the model functions as an Executor (