ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning

Weijie Zhou, Xuangtang Xiong, Ye Tian, Lijun Yue, Xinyu Wu, Wei Li, Chaoyang Zhao, Honghui Dong*, Ming Tang, Jinqiao Wang, and Zhengyou Zhang · 2025

Read ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning by Weijie Zhou, Xuangtang Xiong, Ye Tian, Lijun Yue, Xinyu Wu, Wei Li, Chaoyang Zhao, Honghui Dong*, Ming Tang, Jinqiao Wang, and Zhengyou Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18571.txt.

Introduction

Weijie Zhou and a large team build a robot assistant that knows when to ask for help. Faced with a vague instruction like 'fetch the tool' in a cluttered room, an agent can wander at high physical cost or bother a human at attention cost. Their framework, ESearch-R1, posted to arXiv in December 2025, folds asking, remembering, and navigating into one decision process.

Training uses HC-GRPO, a reinforcement learning method that samples groups of reasoning paths and rewards the ones balancing information gain against travel time and human attention. In the AI2-THOR simulator, ESearch-R1 beats standard ReAct-based agents on success rate while cutting total operating costs by about 50 percent. The results show cost-aware training can align a multimodal model's choices with the real constraints of the physical world.

Catalog record

CREATORWeijie Zhou, Xuangtang Xiong, Ye Tian, Lijun Yue, Xinyu Wu, Wei Li, Chaoyang Zhao, Honghui Dong*, Ming Tang, Jinqiao Wang, and Zhengyou Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18571

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning Weijie Zhou, Xuangtang Xiong, Ye Tian, Lijun Yue, Xinyu Wu, Wei Li, Chaoyang Zhao, Honghui Dong*, Ming Tang, Jinqiao Wang, and Zhengyou Zhang Abstract Multimodal Large Language Models (MLLMs) have empowered embodied agents with remarkable capabilities in planning and reasoning. However, when facing ambiguous natural language instructions (e.g., ”fetch the tool” in a cluttered room), current agents often fail to balance the high cost of physical exploration against the cognitive cost of human interaction. They typically treat disambiguation as a passive perception problem, lacking the strategic reasoning to minimize total task execution costs. To bridge this gap, we propose ESearch-R1, a cost-aware embodied reasoning framework that unifies interactive dialogue (Ask), episodic memory retrieval (GetMemory), and physical navigation (Navigate) into a single decision process. We introduce HC-GRPO (Heterogeneous Cost-Aware Group Relative Policy Optimization). Unlike traditional PPO which relies on a separate value critic, HC-GRPO optimizes the MLLM by sampling groups of reasoning trajectories and reinforcing those that achieve the optimal trade-off between information gain and heterogeneous costs (e.g., navigate time, and human attention). Extensive experiments in AI2-THOR demonstrate that ESearch-R1 significantly outperforms standard ReAct-based agents. It improves task success rates while reducing total operational costs by approximately 50%, validating the effectiveness of GRPO in aligning MLLM agents with physical world constraints. I Introduction Fig. 1: From passive search to proactive interactive disambiguation. In scenes with multiple candidate instances, traditional agents rely on exhaustive navigation and inspection, incurring high movement cost. ESearch-R1 behaves as a collaborative partner: it first narrows down the candidate set via asking the user and retrieving episodic memory, and then executes the shortest path to the uniquely identified goal, thereby reducing both navigation cost and the number of user interruptions. The integration of Multimodal Large Language Models (MLLMs) into robotic systems has catalyzed a shift from rigid instruction following to general-purpose embodied agents capable of long-horizon planning [ 20 , 34 , 40 , 41 , 42 , 43 ] . While these agents demonstrate impressive semantic understanding, a critical misalignment remains between the MLLM’s digital training objective (next-token prediction) and the physical reality of robotic operation [ 1 , 11 , 30 , 9 , 4 , 8 ] . In the digital world, generating a thought is cheap; in the physical world, execution is expensive. A major challenge arises when agents face ambiguity—such as under-specified instructions (e.g., ”Find the cup” in a room with multiple cups) [ 18 , 16 , 21 ] . Standard MLLM agents [ 36 , 31 , 38 ] , often relying on ReAct [ 35 ] or zero-shot prompting, tend to hallucinate a specific target or resort to ”brute-force” physical search [ 22 , 37 ] , sequentially inspecting every candidate. This passive exploration strategy incurs prohibitive costs in terms of time, energy, and wear-and-tear. Conversely, overly cautious agents might spam the user with trivial questions, degrading the human interaction experience. We argue that resolving ambiguity is not merely a perception task but a cost-aware reasoning challenge. A truly intelligent embodied agent should possess ”System 2” thinking capabilities: explicitly weighing the high cost of physical movement against the low cost of internal memory retrieval or the moderate cost of asking a clarification question. The agent must learn a meta-policy that trades off information gain against these heterogeneous costs before taking a step. To achieve this, we introduce ESearch-R1, a novel framework that aligns MLLM reasoning with physical constraints. We formulate the interactive search problem as a Partially Observable Markov Decision Process (POMDP) where dialogue, memory, and navigation are unified actions with distinct cost profiles. The core contribution of ESearch-R1 is its learning paradigm. We propose HC-GRPO (Heterogeneous Cost-Aware Group Relative Policy Optimization). Instead of training a complex value network (Critic) as in standard PPO—which is often unstable for Large Language Models—HC-GRPO leverages group-based sampling. For each query, the agent generates a group of reasoning trajectories (Chain-of-Thought) and actions. The optimization step then reinforces trajectories that maximize a trajectory-level hybrid reward, specifically penalizing high-cost physical actions when cheaper information-seeking alternatives (like referencing episodic memory) were available. This approach enables the emergence of sophisticated disambiguation strategies without explicit rule-coding. The agent learns to ”think before it moves”: checking its internal memory first, asking a targeted question if necessary, and navigating only when the target is uniquely identified. We validate ESearch-R1 on ESearch-Bench, a benchmark built on AI2-THOR representing unstructured semantic environments. Experiments show that our GRPO-tuned agent outperforms strong ReAct baselines (including larger models like Gemini-Pro) by reducing total task costs by \sim 50%, effectively bridging the gap between digital reasoning and physical efficiency. Our main contributions are: • A Cost-Aware Embodied Decision Framework (ESearch-R1) that unifies dialogue, episodic memory, and navigation, enabling agents to actively resolve ambiguity. • Heterogeneous Cost-Aware GRPO (HC-GRPO) , a reinforcement learning algorithm tailored for MLLMs that eliminates the need for a critic network. It effectively optimizes reasoning chains (CoT) to minimize the joint cost of navigation and human interaction. • Emergent Efficient Behaviors. We empirically demonstrate that cost-sensitive RL induces the agent to substitute expensive physical exploration with low-cost cognitive retrieval and strategic dialogue, achieving state-of-the-art efficiency in simulation. II Related Work Our work sits at the intersection of embodied visual search, active perception, and the alignment of MLLM agents via reinforcement learning. We review these areas to highlight the shift from passive instruction following to cost-aware embodied reasoning, positioning ESearch-R1 within the recent advancements in reasoning models. II-A Visual Search and Navigation under Uncertainty Visual search is a cornerstone of embodied AI [ 32 , 27 , 14 , 37 , 21 ] . Standard benchmarks such as Object Navigation (ObjectNav) [ 3 , 23 , 6 ] and Instance Navigation (InstanceNav) [ 24 , 25 ] evaluate an agent’s ability to locate objects based on semantic categories or specific queries. While recent methods leverage hierarchical mapping and frontier-based exploration to improve efficiency, they predominantly model the problem as passive execution. A critical gap in current literature is the handling of ambiguity in natural language instructions (e.g., ”fetch the tool” in a garage with multiple tools) [ 18 , 21 , 16 ] . Most existing baselines operate under a ”trust-and-execute” paradigm, assuming the target is uniquely identifiable or relying on exhaustive, brute-force search when it is not. From a reasoning perspective, this represents a failure of ”System 2” thinking: the agent physically explores the environment to resolve uncertainty that could often be resolved more cheaply via cognitive operations (e.g., retrieving episodic memory). ESearch-R1 addresses this by treating navigation not just as a path-planning problem, but as an information-seeking process where physical movement is a high-cost action to be minimized. II-B Interactive Agents and Active Perception To overcome the limitations of passive search, recent works have explored interactive agents capable of querying humans for assistance [ 19 , 21 , 7 ] . Benchmarks like TEACh [ 19 ] and DialFRED [ 10 ] introduced scenarios where agents must engage in dialogue to complete tasks. Approaches such as Ask-To-Act [ 21 ] use reinforcement learning or uncertainty estimation to decide when to request help. However, prior works typically treat dialogue as an isolated

Read the complete text