R2D2 : Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory

Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi Jonathan May Muhao Chen · 2025

Read R2D2 : Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory by Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi Jonathan May Muhao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12485.txt.

Introduction

Web agents roam complex pages with only partial views of where they have been, so they waste steps and repeat errors. R2D2, a framework by Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi, Jonathan May, and Muhao Chen, tackles this in 2025 with two linked habits. The first builds a replay buffer that lets an agent rebuild a map of visited pages, which cuts steering errors. The second lets it study its past mistakes and revise its approach.

Tested on the WebArena benchmark, the system cut navigation errors by half and tripled task completion rates over earlier methods. The authors suggest that joining memory with honest self-review could help uses like automated customer service and personal digital assistants.

Catalog record

CREATORTenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi Jonathan May Muhao Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R2D2 : Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi Jonathan May Muhao Chen Abstract The proliferation of web agents necessitates advanced navigation and interaction strategies within complex web environments. Current models often struggle with efficient navigation and action execution due to limited visibility and understanding of web structures. Our proposed R2D2 framework addresses these challenges by integrating two paradigms: Remember and Reflect. The Remember paradigm uses a replay buffer that aids agents in reconstructing the web environment dynamically, thus enabling the formulation of a detailed “map” of previously visited pages. This helps in reducing navigational errors and optimizing the decision-making process during web interactions. Conversely, the Reflect paradigm allows agents to learn from past mistakes by providing a mechanism for error analysis and strategy refinement, enhancing overall task performance. We evaluate R2D2 using the WebArena benchmark, demonstrating substantial improvements over existing methods, including a 50% reduction in navigation errors and a threefold increase in task completion rates. Our findings suggest that a combination of memory-enhanced navigation and reflective learning promisingly advances the capabilities of web agents, potentially benefiting various applications such as automated customer service and personal digital assistants. 1 Introduction Web agents—autonomous AI agents designed to navigate and perform natural language-described tasks within web environments—have become increasingly integral to applications such as online customer service Huang et al. (2025) , automated data retrieval Huang et al. (2024) , and personalized digital assistants. 1 1 1 https://www.anthropic.com/news/3-5-models-and-computer-use These agents interact with complex web interfaces to execute user-described tasks, often emulating human actions like clicking buttons, filling forms, and extracting information Shi et al. (2017) ; Liu et al. (2018) ; Yao et al. (2022) ; Zhou et al. (2024b) . Figure 1: Traditional methodologies conceptualize web navigation within the framework of an Unknown MDP. The ReACT agent operates under high uncertainty due to incomplete information regarding the outcomes of its actions, leading to erroneous navigational paths and impeding effective task resolution. R2D2 transforms the task into a Known MDP, improving robustness. Figure 2: This diagram represents various approaches for web agents framed as a search problem, where each node symbolizes a webpage. (a) ReACT: The agent chooses the best immediate actions without any proactive strategy. (b) Tree-search with reflections: the agent investigates different routes by actively navigating websites and allows for reversing direction (shown by dashed arrows). Both a) and b) approaches are Unknown-MDP-based. At each timestep, these agents’ observation space is constrained, which typically results in suboptimal or inefficient results. c) R2D2 : Our proposed framework constructs the search space, leveraging stored state information from a replay buffer. By transforming the task into a Known MDP, R2D2 enhances its ability to navigate and interact with web interfaces. Despite recent advancements in web agents’ capabilities, a persistent challenge remains: agents frequently fail to navigate effectively within intricate web environments as illustrated in Figure 1 . The fundamental challenges associated with previous methodologies are twofold. First, these approaches model web navigation as an Unknown Markov Decision Process (MDP), wherein the agent has limited visibility into the consequences of its actions, often leading to suboptimal performance outcomes. Second, prior methods engage in complex reasoning during the inference phase while observing a stream of experiences, and in their simplest forms, they discard incoming trajectories immediately after a single episode. This rapid forgetting of potentially valuable experiences impedes the agent’s capacity to leverage useful information for future decision-making. Meanwhile, a primary obstacle in enhancing web agent performance lies in navigation-related failures, which account for approximately 60% of their operational errors (as illustrated in Figure 6 ). These failures occur when agents become disoriented within the web environment, preventing them from reaching the target webpages necessary to execute desired tasks. Such navigation inefficiencies significantly hinder the overall effectiveness of web agents. The remaining 40% of errors stem from execution failures and edge cases, where agents either misinterpret user intentions or mishandle specific web elements. Inspired by cognitive studies showing that humans excel at complex tasks by iteratively refining strategies based on feedback Palenciano et al. (2021) ; Zenkri et al. (2024) , as well as by approaches in robotics for structured exploration of unfamiliar spaces Thrun (2002) , we propose the R2D2 ( R emembering, R eplaying, and D ynamic D ecision Making) framework that enhances both navigation and task execution for web agents. Our method transforms the task from an Unknown-MDP into a Known MDP by introducing the Remember Paradigm. It leverages a structured replay buffer of the agent’s experiences that guides the agent to more promising avenues Blundell et al. (2016) ; Schaul et al. (2016) ; Parisotto and Salakhutdinov (2018) ; Savinov et al. (2018) . At a high level, our approach enables the agent to record and recall previously visited pages—essentially constructing a dynamic map of the web environment—and then leverage this knowledge to refine its strategies. By converting the agent’s experience into a well-organized search space, we empower it to identify reliable navigation routes to target resources rather than re-deriving them from scratch during inference. This reduces the computational overhead at test time and helps avoid repetitive or unproductive exploration. To enable continual improvement based on both successes and failures, R2D2 incorporates the Reflect Paradigm. Previous efforts in this domain often focus narrowly on immediate, execution-level errors and struggle with more pervasive navigation challenges Shinn et al. (2023) ; Pan et al. (2024) ; Wang et al. (2024a) . In contrast, our method leverages the refined, known search space described earlier to minimize navigational missteps, allowing the reflection mechanism to operate more effectively on remaining execution problems. By reducing the burden of basic wayfinding, the agent’s reflective capabilities can more efficiently identify and correct subtle issues, ultimately leading to a higher overall success rate on complex web tasks. Our proposed method diverges from traditional techniques by providing a more comprehensive and structured representation of the agent’s historical experiences. Instead of simply recalling past states Agashe et al. (2024) or relying on on-the-fly reasoning Koh et al. (2024) ; Zhou et al. (2024a) , our approach organizes the agent’s accumulated experiences into a coherent and reusable resource that effectively guides future decisions. We evaluate our approach using the WebArena benchmark Zhou et al. (2024b) , where it achieves substantial gains compared to baseline models, including approximately a 50% reduction in navigation errors and a threefold increase in overall task completion rates. Moreover, R2D2 outperforms state-of-the-art methods by 17%, thereby demonstrating a more robust and informed capability for executing complex web-based tasks. 2 Related Works Enhancing Web Agents. Existing research has shown that language models, without intervention, struggle to express linguistic intent in formal instruction that can control an extra-linguistic environment, such as web site navigation Shi et al. (2017) ; Liu et al. (2018) ; Yao et al. (2022) ; Zhou et al. (2024b) ; Deng et al. (2023) . This inadequacy stems primarily from the intrinsic challenges associated with perception, strategic planning, and task execution in the intricate web environments. To mitigate these challenges, enhancements to web agents have been characterized as one of three principal strategies: (1) Perception Alignment : This strategy aims to augment agents’ capabilities

Read the complete text