RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation

Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunfeng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, Long Chen · 2025

Read RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation by Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunfeng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, Long Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12436.txt.

Introduction

Xiaoshuai Hao and eight co-authors introduce RoboAfford-plus-plus, a large dataset for teaching robots what parts of a scene they can act on. Robots that manipulate objects and move through space need more than object labels: they need grasp points, placement spots and free areas for motion, known as affordances. Vision-language models often miss those fine action details because their training data lack such annotations.

The dataset pairs a very large image set with millions of question-answer annotations made with generative AI help. It covers object affordance recognition and related spatial tasks for both manipulation and navigation. The aim is to give embodied systems the fine-grained grounding they need to interact with the physical world.

Catalog record

CREATORXiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunfeng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, Long Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12436

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunfeng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, Long Chen Abstract Robotic manipulation and navigation are fundamental capabilities of embodied intelligence, enabling effective robot interactions with the physical world. In manipulation, predicting precise interactive positions is essential for grasping and placing objects. In navigation, find the target and understanding the traversable free space are crucial for safe movement. Achieving these capabilities requires a cohesive understanding of the environment, including object recognition to localize target objects, object affordances to identify potential interaction areas and spatial affordances to discern optimal areas for both object placement and robot movement. While Vision-Language Models (VLMs) excel at high-level task planning and scene understanding, they often struggle to infer actionable positions for physical interaction, such as functional grasping points and permissible placement regions. This limitation stems from the lack of fine-grained annotations for object and spatial affordances in their training datasets. To tackle this challenge, we introduce RoboAfford++ , a generative AI-enhanced dataset for multimodal affordance learning for both robotic manipulation and navigation. Our dataset comprises 869,987 images paired with 2.0 million question answering (QA) annotations, covering three critical tasks: object affordance recognition to identify target objects based on attributes and spatial relationships, object affordance prediction to pinpoint functional parts for manipulation, and spatial affordance localization to identify free space for object placement and robot navigation. Complementing this dataset, we propose RoboAfford-Eval , a comprehensive benchmark for assessing affordance-aware prediction in real-world scenarios, featuring 338 meticulously annotated samples across the same three tasks. Extensive experimental results reveal the deficiencies of existing VLMs in affordance learning, while fine-tuning on the RoboAfford++ dataset significantly enhances their ability to reason about object and spatial affordances, validating the dataset’s effectiveness. The dataset, benchmark and evaluation code will be made publicly available to facilitate future research. Project website: https://roboafford-dataset.github.io/ . I INTRODUCTION Enabling robots to physically interact with 3D environments through manipulation and navigation presents a fundamental challenge in embodied AI, requiring advanced perception, reasoning, and adaptability to handle real-world complexities [ 26 , 20 , 21 , 51 ] . Recent advancements in VLMs [ 15 , 22 , 11 , 4 , 23 , 47 , 46 , 2 ] have significantly improved robots’ perceptual and reasoning capabilities, facilitating progress in scene understanding and task planning. These models have shown considerable potential in various embodied tasks, including vision-language navigation [ 67 , 59 , 7 , 62 , 60 , 58 , 52 , 61 , 31 , 17 ] , embodied manipulation [ 55 , 53 , 45 , 19 , 57 , 44 , 46 , 65 ] , and embodied reasoning [ 47 , 64 , 43 ] . However, despite these advancements in high-level planning and semantic understanding, existing VLMs still encounter significant challenges in achieving fine-grained interaction understanding for both manipulation and navigation scenarios. Fig. 1: Overview of the RoboAfford++ Dataset. It encompasses three key capabilities: object affordance recognition, object affordance prediction, and spatial affordance localization for both manipulation and navigation tasks. In robotic manipulation and navigation, VLMs are required to ground target objects or their parts from language instructions while identifying suitable regions for placement or movement. This involves three core capabilities: (1) Object Affordance Recognition : identifying objects based on attributes such as category, color, size, and spatial relations; (2) Object Affordance Prediction : localizing functional parts of objects to support specific actions, such as the handle of a teapot for grasping; and (3) Spatial Affordance Localization : detecting vacant areas in the scene for object placement and robot navigation, such as empty space in shelves for storing items. Despite recent progress [ 54 , 42 , 44 ] , relatively few works have achieved a holistic integration of object and spatial affordances that adequately supports both manipulation and navigation tasks. While some VLMs [ 54 , 42 ] can coarsely localize objects or assess spatial compatibility through coordinates, they often fail to perform fine-grained localization at the part level, such as identifying functional components of objects. Furthermore, the models are specifically trained for manipulation tasks, resulting in a narrow focus that limits their generalization across diverse manipulation and navigation scenarios in real world. To address these challenges, we present RoboAfford++ , a generative AI-enhanced large-scale dataset with dense, affordance-aware annotations for robotic manipulation and navigation. It contains 869,987 images and 2.0 million QA pairs, unifying object and spatial affordances to support interaction-centric learning across both domains.. As shown in Fig. 1 , RoboAfford++ aims to equip VLMs with affordance reasoning capabilities essential for embodied interaction. The dataset enables models to ground target objects and predict part-level affordances for interaction, and identify spatial affordances for both placement and navigation. By employing generative AI techniques for data augmentation and annotation, RoboAfford++ specifically addresses the challenge of translating high-level instructions into actionable positions for manipulation and feasible areas for navigation. Additionally, we introduce RoboAfford-Eval , a rigorously annotated benchmark featuring 338 evaluation questions. RoboAfford-Eval encompasses three tasks: (1) object affordance recognition to assess grounding ability, (2) object affordance prediction to evaluate functional part understanding, and (3) spatial affordance localization to test free space detection. Extensive experiments demonstrate that fine-tuning on the RoboAfford++ dataset significantly enhances the model’s affordance reasoning capabilities for both manipulation and navigation tasks. Our contributions are summarized as follows: TABLE I: Comparison of existing affordance datasets. Obj-Aff : Object Affordance. Spa-Aff : Spatial Affordance. Dataset Domain Format Obj-Aff Spa-Aff #Spatial Relations #Images #QAs UMD [ 36 ] Tabletop 2D Mask ✓ ✗ - 30.0K - IIT-AFF [ 37 ] Generic 2D Mask ✓ ✗ - 8.8K - PAD [ 34 ] Generic 2D Mask ✓ ✗ - 4.0K - PADv2 [ 56 ] Generic 2D Mask ✓ ✗ - 30.0K - AGD-20K [ 35 ] Generic 2D Heatmap ✓ ✗ - 26.1K - 3DOI [ 39 ] Generic, indoor 2D Mask, point ✓ ✗ - 10.0K - RoboPoint [ 54 ] Generic, tabletop 2D Point ✗ ✓ 14 - 1.4M RoboSpatial [ 42 ] Indoor, tabletop 2D Point ✗ ✓ 6 1.0M 3.0M RoboAfford++ (Ours) Generic, indoor, tabletop 2D Point ✓ ✓ 20 870.0K 2.0M • We introduce RoboAfford++ , a novel generative AI-enhanced dataset that unifies object and spatial affordances for both robotic manipulation and navigation. It comprises 2.0 million question answering pairs with precise 2D point annotations, focusing on three key capabilities: object affordance recognition, object affordance prediction, and spatial affordance localization. This comprehensive dataset fills the critical gap in fine-grained affordance annotations necessary for precise physical interactions. • We present the RoboAfford-Eval benchmark, featuring 338 manually annotated questions designed to systematically evaluate VLMs on three core tasks: object affordance recognition, object affordance prediction, and spatial affordance localization. This benchmark establishes criteria to assess both object-level and scene-level affordance understanding. • Extensive experiments reveal the limitations of existing VLMs in affordance learning, while fine-tuning on the RoboAfford++ dataset significantly enhances their affordance prediction for both manipulation and navigation scenarios, confirming its effectiveness and practical value. II RELATED WORK Datasets for Affordance Learning Affordance learning is a key research area in computer vision and robotics, which focuses on

Read the complete text