H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows

Harry Zhang, Luca Carlone · 2025

Read H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows by Harry Zhang, Luca Carlone (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21769.txt.

Introduction

Harry Zhang and Luca Carlone teach machines the full grammar of how people use objects, without paying humans to label any of it. An affordance is more than a contact patch. People also face objects from preferred sides and occupy favored zones around them, like standing before a microwave rather than behind it. Datasets for such subtleties are costly hand-made affairs.

Their framework, H2OFlow, learns contact, orientation, and spatial occupancy together from synthetic scenes produced by 3D generative models. The representation is a dense flow over point clouds, learned by a diffusion process, from which rich three-dimensional affordances can be read off. In quantitative and qualitative tests, the system transfers to real objects and beats methods that relied on manual annotation or mesh-based representations.

Catalog record

CREATORHarry Zhang, Luca Carlone
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows Harry Zhang, Luca Carlone Abstract Understanding how humans interact with the surrounding environment, and specifically reasoning about object interactions and affordances, is a critical challenge in computer vision, robotics, and AI. Current approaches often depend on labor-intensive, hand-labeled datasets capturing real-world or simulated human-object interaction (HOI) tasks, which are costly and time-consuming to produce. Furthermore, most existing methods for 3D affordance understanding are limited to contact-based analysis, neglecting other essential aspects of human-object interactions, such as orientation ( e.g. , humans might have a preferential orientation with respect certain objects, such as a TV) and spatial occupancy ( e.g. , humans are more likely to occupy certain regions around an object, like the front of a microwave rather than its back). To address these limitations, we introduce H2OFlow , a novel framework that comprehensively learns 3D HOI affordances —encompassing contact, orientation, and spatial occupancy— using only synthetic data generated from 3D generative models. H2OFlow employs a dense 3D-flow-based representation, learned through a dense diffusion process operating on point clouds. This learned flow enables the discovery of rich 3D affordances without the need for human annotations. Through extensive quantitative and qualitative evaluations, we demonstrate that H2OFlow generalizes effectively to real-world objects and surpasses prior methods that rely on manual annotations or mesh-based representations in modeling 3D affordance. Interactive 3D visualization, code, and data will be available at this website . 1 Introduction The rapid advancement of AI and robotics demands next-generation agents that can perceive and interact with the world as seamlessly as humans do. A key aspect of human intelligence is the innate ability to recognize the functionalities offered by objects and environments —allowing us to effortlessly adapt to unstructured settings like homes. For AI agents to achieve similar generalization, they must learn how to interact with objects based on their intended purpose —a concept known as affordance . First introduced by psychologist James Gibson Gibson 2014 , the concept of affordance has become an important topic for advancing AI and robot capabilities in our daily life. A plethora of studies have been conducted on affordances for visual recognition Hou et al. 2021 , Hong et al. 2023 , action prediction Roy & Fernando 2021 , Chen et al. 2023 , and functionality understanding Li et al. 2023a , Zhang et al. 2023 , Kim et al. 2024 . Understanding affordances through the lens of human-object interactions (HOIs) also offers a compelling approach for teaching AI agents. By observing how humans manipulate and interact with objects, we can extract rich cues about objects’ functionality, thus enabling a broader set of interactions for AI agents. However, prior work in HOI affordance learning has largely focused on contact-based affordances, which is a restrictive subset of all possible affordances. For instance, recent methods estimate contact scores from RGB images Bahl et al. 2023 , Bahl et al. 2022 , Li et al. 2023a , 3D point clouds Chu et al. 2025 , Yang et al. 2023 , or human models Hassan et al. 2021 , by relying on densely annotated human-object contact labels. This manual supervision is not only labor-intensive but also fails to generalize to novel objects and broader classes of interaction. Figure 1: H2OFlow learns comprehensive affordances from synthetic 3D HOI data generated by 3D generative models using a novel representation. The learned affordance captures contact, orientational, and occupancy information based on input object point clouds. We observe that human-object interactions (HOIs) involve 3D spatial relationships beyond simple contact. For example, human faces, torsos, and arms often maintain characteristic distances and orientations relative to objects, with natural variations across interactions. For instances, humans would grasp different tools with different hand configurations: a hammer is typically held at a specific distance from its head, with the wrist angled to allow effective striking, while a pen is gripped closer to the tip for finer control. A complete understanding of affordances in HOIs should incorporate these geometric patterns, including relative positioning and orientational tendencies. A recent work by Kim et al. 2024 introduces the concept of comprehensive affordance , which captures these relationships probabilistically. Instead of binary contact labels, their method models a distribution over possible 3D spatial and orientational relations between every pair of object and human surface points. This approach generalizes affordance reasoning beyond contact, enabling finer-grained understanding of interaction geometry. As shown in Kim et al. 2024 , learning comprehensive affordances in HOIs typically relies on synthetic RGB images uplifted to 3D using 2D-to-3D techniques. However, this approach requires intricate masking methods to achieve high-quality results, introducing multiple potential failure modes. Furthermore, the learned affordances often fail to generalize to novel real-world objects, and the dependency on well-defined watertight meshes for better-quality affordance computation severely limits real-world applicability. To address these challenges, we leverage recent advances in 3D generative models for HOIs Li et al. 2024a , Diller & Dai 2024 , Peng et al. 2023 . Our key innovation is a pipeline that directly generates plausible 3D HOI samples using generative models, eliminating the need for error-prone 2D-to-3D uplifting. To ensure generalization to novel geometries, we subsample points from the generated data and employ dense diffused flows Eisner et al. 2022 —a technique proven effective for modeling multi-modality— to reconstruct 3D humans from the HOI samples. For comprehensive affordance learning, we introduce a novel probabilistic formulation operating directly on human-object point cloud pairs, circumventing the need for a watertight mesh. This culminates in H uman- O bject Flow ( H2OFlow ), a framework for learning rich affordance knowledge in HOIs (Fig. 1 ). Our key contributions are: 1. A point-cloud-based affordance representation that efficiently captures both explicit contact and implicit non-contact interaction patterns in HOIs from raw point clouds inputs. 2. A synthetic data generation and learning pipeline, which leverages 3D generative models and dense diffused flows, that learns flexible affordances from synthetic 3D point clouds. 3. Extensive quantitative and qualitative experiments demonstrating the effectiveness and practical utility of the learned affordances on both synthetic datasets and real-world data. 2 Related Work Affordance Learning. First introduced in Gibson 2014 , affordance learning has emerged as a critical capability for AI and robotic systems. Modern approaches focus on enhancing agents’ ability for better visual recognition Hou et al. 2021 , Hong et al. 2023 , action prediction Roy & Fernando 2021 , Chen et al. 2023 , functionality understanding Li et al. 2023a , Eisner et al. 2022 , Kim et al. 2024 , mimicking scene-conditioned human-object and hand-object interactions Bhatnagar et al. 2022 , Lu et al. 2022 , Nguyen et al. 2024 , Jiang et al. 2022 , Huang et al. 2022 , Jiang et al. 2022 , Petrov et al. 2023 , Hassan et al. 2021 , Zhang et al. 2022 . With the advances of LLMs, more works have been proposed to explore open-vocabulary affordances in point clouds Nguyen et al. 2024 , Chu et al. 2025 . However, most works focus exclusively on contact-based affordances, neglecting crucial spatial and orientational aspects of interactions. Moreover, the requirement of manual labeling of the contact regions Do et al. 2018 , Jian et al. 2023 , Tripathi

Read the complete text