ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
Liu Dai, Haina Wang, Weikang Wan, Hao Su · 2025
Read ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making by Liu Dai, Haina Wang, Weikang Wan, Hao Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20726.txt.
Introduction
Liu Dai, Haina Wang, Weikang Wan, and Hao Su introduce ManiTaskGen, a system that automatically generates manipulation tasks for training and testing embodied agents. Building robots that can accomplish arbitrary tasks is a core goal of embodied AI. Yet training and evaluation are usually confined to a few scenes with restricted instructions, and benchmarks rely on manual annotation of limited task sets.
ManiTaskGen explores the full spectrum of feasible tasks within any given scene. It generates comprehensive, diverse mobile manipulation tasks, both process-based instructions like moving an object from one place to another and outcome-based ones like clearing the table. Applied to simulated and real-world scenes, the generated tasks proved valid and diverse. The authors use them to build benchmarks automatically and to evaluate agents built on existing vision-language models. They also propose a simple method that uses the generated tasks to improve those agents.
Catalog record
| CREATOR | Liu Dai, Haina Wang, Weikang Wan, Hao Su |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 206 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making Liu Dai, Haina Wang, Weikang Wan, Hao Su Abstract Building embodied agents capable of accomplishing arbitrary tasks is a core objective towards achieving embodied artificial general intelligence (E-AGI). While recent work has advanced such general robot policies, their training and evaluation are often limited to tasks within specific scenes, involving restricted instructions and scenarios. Existing benchmarks also typically rely on manual annotation of limited tasks in a few scenes. We argue that exploring the full spectrum of feasible tasks within any given scene is crucial, as they provide both extensive benchmarks for evaluation and valuable resources for agent improvement. Towards this end, we introduce ManiTaskGen, a novel system that automatically generates comprehensive, diverse, feasible mobile manipulation tasks for any given scene. The generated tasks encompass both process-based, specific instructions (e.g., "move object from X to Y") and outcome-based, abstract instructions (e.g., "clear the table"). We apply ManiTaskGen to both simulated and real-world scenes, demonstrating the validity and diversity of the generated tasks. We then leverage these tasks to automatically construct benchmarks, thoroughly evaluating the embodied decision-making capabilities of agents built upon existing vision-language models (VLMs). Furthermore, we propose a simple yet effective method that utilizes ManiTaskGen tasks to enhance embodied decision-making. Overall, this work presents a universal task generation framework for arbitrary scenes, facilitating both benchmarking and improvement of embodied decision-making agents. 1 Introduction Consider an embodied agent endowed with robust primitive skills for mobile manipulation: the ability to navigate to any accessible location, grasp any movable object, and place it wherever it fits. A fundamental question then arises: what is the full extent of the task space this agent can successfully address in a given environment? This space appears infinite. Despite this immense potential task space, recent efforts have focused on developing general-purpose embodied agents [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] capable of completing any feasible task within it. A promising direction leverages VLMs [ 8 , 9 , 10 ] for high-level decision-making, followed by either the explicit composition of skill primitives [ 11 , 12 , 13 ] or the implicit integration with underlying action modules to predict joint-level actions—the latter thread is commonly referred to as Vision-Language-Action (VLA) models [ 14 , 15 , 5 ] . To evaluate such agents, numerous benchmarks [ 16 , 17 , 18 , 19 , 20 ] have also been proposed to assess their decision-making capabilities. Figure 1: Framework Overview. ManiTaskGen is a universal system that generates a comprehensive set of feasible mobile manipulation tasks given arbitrary scene. These tasks facilitate automatic benchmarking and the improvement of embodied decision-making agents. Despite these efforts, existing work often faces significant limitations regarding the tasks used in both policy training and benchmark design. Common practice [ 21 , 13 , 22 , 20 ] involves manually selecting a few scenes and authoring a limited number of scene-specific tasks related to objects within those scenes, which serve as training or testing objectives for embodied agents. Such manual approaches are labor-intensive and require considerable human effort to design suitable tasks, resulting in prohibitively high scaling costs and making it nearly impossible to generalize task creation to the diverse range of scenes encountered in both simulated and real-world environments. This fundamental limitation creates a substantial gap between current agents and the ultimate goal of E-AGI: while the aim is an agent capable of universal generalization across diverse scenes and tasks, existing works are confined to training and evaluating the agents within limited scenarios and task variations. To address these limitations, we present ManiTaskGen, a universal mobile manipulation task generator for arbitrary scenes. Given scene information (e.g., object poses, object bounding boxes, object mesh models, etc.), it automatically generates a comprehensive and diverse collection of feasible mobile manipulation tasks that are logically near-exhaustive for that specific scene. We ensure logical comprehensiveness by grounding task generation in a systematic analysis of the fundamental agent-environment interaction cycle, as conceptually illustrated in Fig. 2 . Figure 2: Agent-Env Interaction Cycle. This cycle provides a fundamental depiction of how an embodied agent executes a task within a scene. Accordingly, we categorize all tasks into two distinct types: process- and outcome-based tasks. This cycle, where executing atomic actions (e.g., single object relocation) updates the scene state and determines subsequent available actions, inherently defines the space of all possible tasks. Based on this inherent structure, we rigorously categorize all possible tasks into two principal types: process-based tasks , capturing action sequences or trajectories (e.g., "move object A from X to Y"), and outcome-based tasks , representing reachable target states (e.g., "make the table clean"). ManiTaskGen employs distinct strategies for generating each type. Process-based tasks are generated by explicitly sampling and composing atomic action sequences, which are derived from a novel Receptacle-Aware 3D Scene Graph encoding all objects and fine-grained potential placements within the scene. Outcome-based tasks are generated using a hybrid template-based approach combined with VLM voting mechanism to produce diverse descriptions of plausible target states. We assess the validity and diversity of the generated tasks by applying ManiTaskGen to both simulated environments (e.g., ReplicaCAD [ 23 ] , AI2THOR [ 24 ] ) and real-world scenes (e.g., SUN-RGBD [ 25 ] ). Furthermore, we propose a framework for automatically constructing benchmarks in simulators using the generated tasks to assess embodied agent decision-making capabilities, and conduct extensive evaluations of existing VLMs using this benchmark. Finally, we design an improvement method based on Inference-time Reinforcement Learning [ 26 , 27 ] to leverage ManiTaskGen tasks for enhancing the decision-making abilities of existing VLM agents. In summary, this work makes the following contributions: (1) We propose ManiTaskGen, a universal system for generating comprehensive and diverse mobile manipulation tasks for arbitrary scenes. (2) Leveraging the automatically constructed benchmarks based on the generated tasks, we conduct an extensive evaluation of the embodied decision-making capabilities of current VLMs. (3) We demonstrate the utility of the ManiTaskGen tasks for enhancing embodied decision-making in current VLMs through a proposed inference-time RL method. 2 Related Work Task Generation for Embodied Agents. Recent efforts have explored task generation for digital agents [ 28 , 29 , 30 , 31 ] and augmenting RL objectives [ 32 , 33 ] . Among these, ALFRED [ 34 ] is the most relevant to our work. It combines a procedural task planner with human-annotated task directives. However, ALFRED focuses on expanding task trajectories rather than task definitions and lacks a comprehensive coverage of both process- and outcome-based mobile manipulation tasks. Moreover, it is limited to specific scenes. In contrast, ManiTaskGen aims to generate a diverse and comprehensive set of tasks for arbitrary scenes, emphasizing scalability and variability in task formulation. Datasets and Benchmarks for Embodied Agents. Numerous datasets and benchmarks have been proposed for training [ 35 , 24 , 36 , 3 , 37 ] and evaluating [ 21 , 34 , 37 ] embodied agents, including those tailored to LLM/VLM-based decision-making agents [ 17 , 18 , 38 , 22 ] . A common limitation of these benchmarks is their reliance on manually annotated tasks confined to a finite