Enabling Extensible Embodied Capabilities with Tools

Xueyang Zhou 1 , Zijia Wang 1 , Qianjiang Li 2 , Yibo Hu 1 , Guiyao Tie 1 , Li Wan 1 , Yidan Liu 3 , Pan Zhou 1,∗ , Lichao Sun 4 , Yongchao Chen 5,∗ 1 Huazhong University of Science and Technology 2 Hebei University of Technology, 3 Tianjin University, 4 Lehigh University 5 College of AI, Tsinghua University {d202480819, m202572276, u202413536, tgy}@hust.edu.cn {u202315903, panzhou}@hust.edu.cn [email protected], [email protected] [email protected], [email protected] ∗ Corresponding authors: Pan Zhou and Yongchao Chen · 2026

Read Enabling Extensible Embodied Capabilities with Tools by Xueyang Zhou 1 , Zijia Wang 1 , Qianjiang Li 2 , Yibo Hu 1 , Guiyao Tie 1 , Li Wan 1 , Yidan Liu 3 , Pan Zhou 1,∗ , Lichao Sun 4 , Yongchao Chen 5,∗ 1 Huazhong University of Science and Technology 2 Hebei University of Technology, 3 Tianjin University, 4 Lehigh University 5 College of AI, Tsinghua University {d202480819, m202572276, u202413536, tgy}@hust.edu.cn {u202315903, panzhou}@hust.edu.cn [email protected], [email protected] [email protected], [email protected] ∗ Corresponding authors: Pan Zhou and Yongchao Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26637.txt.

Introduction

Xueyang Zhou and nine others move robot skills out of one big policy and into callable tools. Unified policies struggle because perception, reasoning, planning, and control differ in kind and resist neat modular learning. Their Embodied Tool Protocol standardizes tool registration, discovery, calls, and runs. It draws on over 100 checked tools and a bench covering tool need, choice, use, and chaining. Tests show gains averaging 31 percent on one bench and 36 percent on another, strong for thought and sight but weak for action.

Catalog record

CREATORXueyang Zhou 1 , Zijia Wang 1 , Qianjiang Li 2 , Yibo Hu 1 , Guiyao Tie 1 , Li Wan 1 , Yidan Liu 3 , Pan Zhou 1,∗ , Lichao Sun 4 , Yongchao Chen 5,∗ 1 Huazhong University of Science and Technology 2 Hebei University of Technology, 3 Tianjin University, 4 Lehigh University 5 College of AI, Tsinghua University {d202480819, m202572276, u202413536, tgy}@hust.edu.cn {u202315903, panzhou}@hust.edu.cn [email protected], [email protected] [email protected], [email protected] ∗ Corresponding authors: Pan Zhou and Yongchao Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110881
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26637

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enabling Extensible Embodied Capabilities with Tools Xueyang Zhou 1 , Zijia Wang 1 , Qianjiang Li 2 , Yibo Hu 1 , Guiyao Tie 1 , Li Wan 1 , Yidan Liu 3 , Pan Zhou 1,∗ , Lichao Sun 4 , Yongchao Chen 5,∗ 1 Huazhong University of Science and Technology 2 Hebei University of Technology, 3 Tianjin University, 4 Lehigh University 5 College of AI, Tsinghua University {d202480819, m202572276, u202413536, tgy}@hust.edu.cn {u202315903, panzhou}@hust.edu.cn [email protected], [email protected] [email protected], [email protected] ∗ Corresponding authors: Pan Zhou and Yongchao Chen Abstract Most existing embodied intelligence methods formulate perception, reasoning, planning, and control within a unified parameterized policy. Yet these capabilities are inherently hierarchical and heterogeneous, making them difficult to reliably learn and modularize within a single model. We propose a capability externalization approach that decouples heterogeneous capabilities into independently optimized tools, dynamically invoked at inference time. To this end, we introduce E mbodied T ool P rotocol ( ETP ), a standardized protocol for embodied tool registration, discovery, invocation, and execution, and curate 100+ validated tools spanning perception, cognition, reasoning, and execution as the tool base. Building on this, we construct EmbodiedToolBench to evaluate both whether tool augmentation improves embodied performance and how well current models use tools across tool-necessity recognition, tool selection, tool execution, and tool-chain composition. Experiments across simulation and real-world platforms confirm that capability externalization consistently improves embodied performance (avg. gain 31% on EB-ALFRED and 36% on EB-Navigation), yet reveal a clear boundary: gains are substantial for cognition and perception but are limited for execution-type capabilities. Moreover, our analysis reveals that knowing when, which, and how to invoke tools remains a persistent challenge across all models, thereby highlighting embodied tool competence as a critical direction for future research. GitHub: https://github.com/Zxy-MLlab/EmbodiedTool Webpage: https://racingemperor.github.io/manip-tool-hub/ 1 Introduction Embodied intelligence requires agents to perceive, reason, and act in physical environments through coupled perception, planning, and control [ 50 , 23 , 37 ] . Recent advances have centered on parameterized embodied policies, including vision-language-action models [ 54 , 6 , 4 ] and hierarchical vision-language systems [ 1 ] , which have been strengthened through improved spatial grounding, richer perception, and enhanced reasoning [ 40 , 2 , 11 ] . Despite substantial gains on standard benchmarks, these monolithic architectures underperform on long-horizon, compositional, and safety-critical tasks, a gap that points to a structural, rather than a capacity, limitation. The core difficulty is that embodied decision-making is inherently hierarchical and heterogeneous [ 50 , 43 , 23 , 26 ] : high-level planning and low-level control differ fundamentally in their functional roles and optimization objectives. Encoding both within a single shared parameterization introduces three intrinsic limitations. Opaque coupling : capabilities are jointly encoded without clear attribution, preventing selective invocation or diagnosis. Capability isolation : competencies acquired by one model remain siloed and cannot be transferred or reused across systems. Entangled optimization : improving one capability risks degrading others due to conflicting gradient signals over shared parameters. Together, these limitations motivate tool-based embodied intelligence [ 48 , 38 , 18 ] , an approach that externalizes specific capabilities as callable, independently optimizable tools that can be registered, discovered, and composed across models and tasks. Yet research on tool-based embodied intelligence remains fragmented, lacking a unifying framework to guide design and evaluation. Three fundamental questions remain open: (1) Protocol: how should embodied capabilities be represented, registered, and invoked in a principled, extensible manner? (2) Efficacy: can current models reliably use external tools, and how much does tool augmentation improve embodied performance in practice? (3) Externalization boundary: which capabilities benefit from decoupling into tools, and where does this benefit diminish? To address these questions, we establish a unified framework centered on the Embodied Tool Protocol (ETP), which provides a principled foundation for embodied tool registration, discovery, invocation, and execution. Grounded in ETP, we curate a tool base of 100+ validated tools spanning perception and grounding, cognition and state modeling, reasoning and planning, and execution and control. Building on this, we construct EmbodiedToolBench to systematically evaluate the tool-use competence of current models across tool-need recognition, tool selection, tool execution, and tool-chain composition. Through systematic experiments across simulation and real-world platforms, we characterize the externalization boundary and identify the key bottlenecks that current models face in embodied tool use. • We introduce ETP , a standardized protocol for embodied tool registration, discovery, invocation, and execution, and for the first time curate 100+ validated tools spanning perception, cognition, reasoning, and execution as a unified tool base. • We present EmbodiedToolBench , a benchmark evaluating embodied tool use competence across tool-need recognition, tool selection, tool execution, and tool-chain composition in planning, navigation, and manipulation scenarios. • Through extensive experiments across simulation and real-world platforms, we demonstrate that tool augmentation consistently improves embodied task performance, and reveal systematic bottlenecks in how current models autonomously use embodied tools. 2 Preliminary 2.1 Embodied Agent We formulate embodied tasks as sequential decision-making problems. Let o_{t}\in\mathcal{O} be the observation at timestep t , l\in\mathcal{L} the language instruction, and \tau_{t}:=(o_{0},a_{0},\dots,o_{t}) the interaction history, with history space \mathcal{H}_{t}:=\mathcal{O}^{\leq t}\times\mathcal{A}^{<t} . A monolithic embodied agent defines a single policy: \pi_{\theta}:\;\mathcal{H}_{t}\times\mathcal{L}\;\longrightarrow\;\Delta(\mathcal{A}),\qquad a_{t}\sim\pi_{\theta}(\cdot\mid\tau_{t},\,l), (1) where \theta jointly encodes all capabilities at every decision step. Let \mathcal{C}=\{c_{1},\ldots,c_{K}\} be the full capability set required across tasks (e.g. low-level control, spatial reasoning, memory), and \mathcal{C}(d)\subseteq\mathcal{C} the subset demanded by task d . The agent maximizes the expected discounted return via: \mathcal{J}_{\mathrm{mono}}(\theta)\;:=\;\mathbb{E}_{d\sim p(d)}\!\left[\mathbb{E}_{\tau\sim p(\tau\mid\pi_{\theta},\,d)}\!\left[\,\sum_{t=0}^{T}\gamma^{t}r_{t}\right]\right],\;\gamma\in(0,1],\quad\theta^{*}=\operatorname*{arg\,max}_{\theta}\,\mathcal{J}_{\mathrm{mono}}(\theta). (2) Because \theta must handle all tasks in \mathcal{D} , the heterogeneous capabilities \mathcal{C} are entangled within a single parameter set, regardless of which subset \mathcal{C}(d) is actually needed per step. 2.2 Embodied Agent with Tools To overcome capability entanglement, we externalize each capability as an independently optimized tool invoked on demand. Tool collection. Define the tool collection \mathcal{Z}:=\{z_{m}(\,\cdot\,;\,\phi_{m})\}_{m=1}^{M} , where each tool z_{m} , parameterized by \phi_{m} , realizes a capability subset \mathcal{C}(z_{m})\subseteq\mathcal{C} . Tools are optimized independently of one another, and their collective coverage \mathcal{C}_{\mathcal{Z}}:=\bigcup_{m=1}^{M}\mathcal{C}(z_{m}) replaces the entangled encoding inside \theta . Tool-augmented decision process. At each step t , the agent selects a tool g_{t}\in\bar{\mathcal{Z}}:=\mathcal{Z}\cup\{\perp\} (where \perp denotes invoking no tool), queries it with a generated query q_{\theta}(\tau_{t},l,g_{t}) , and conditions its action on the returned observation y_{t} . Formally, the three-stage transition is: g_{t}\sim\mu_{\theta}(\cdot\mid\tau_{t},\,l),\qquad y_{t}\;:=\;\mathcal{T}_{g_{t}}\!\left(q_{\theta}(\tau_{t},\,l,\,g_{t})\right),\qquad a_{t}\sim\pi_{\theta}(\cdot\mid\tau_{t},\,l,\,y_{t}). (3) Bi-level optimization. Learning decomposes into two decoupled objectives. The lower-level objective trains each tool independently on its designated capability dataset \mathcal{D}_{m} : \mathcal{J}_{\mathrm{tool}}(\phi_{m})\;:=\;\mathcal{L}_{m}(\phi_{m};\,\mathcal{D}_{m}),\qquad\phi_{m}^{*}\;=\;\operatorname*{arg\,min}_{\phi_{m}}\;\mathcal{J}_{\mathrm{tool}}(\phi_{m}),\quad m\in[M]. (4) The upper-level objective trains the orchestration policy \theta to maximize expected return over the full task distribution: \mathcal{J}_{\mathrm{orch}}(\theta)\;:=\;\mathbb{E}_{d\sim p(d)}\!\left[\mathbb{E}_{\tau\sim p_{\theta}(\tau\mid d)}\!\left[\,\sum_{t=0}^{T}\gamma^{t}r_{t}\right]\right],\qquad\theta^{*}\;=\;\operatorname*{arg\,max}_{\theta}\;\mathcal{J}_{\mathrm{orch}}(\theta). (5) Each tool \phi_{m}^{*} is optimized independently for its designated capability ( Equation 4 ), while \theta^{*} learns to orchestrate the tool collection for downstream task completion ( Equation 5 ). This yields two key advantages: (i) capability-specific optimization without cross-task interference, and (ii) on-demand invocation that avoids embedding all capabilities into \theta at every step. 3 Embodied Tools Figure 1 : Overview of the EmbodiedTool . 3.1 Embodied Tool Protocol Heterogeneous embodied capabilities, ranging from low-level motor control to high-level spatial reasoning, differ in their interfaces, parameterization, and execution requirements. Composing them within a single agent therefore requires a shared abstraction that decouples what a capability does from how the agent invokes it.

Read the complete text