Affostruction: 3D Affordance Grounding with Generative Reconstruction

Chunghyun Park, Seunghyeon Lee, Minsu Cho · 2026

Read Affostruction: 3D Affordance Grounding with Generative Reconstruction by Chunghyun Park, Seunghyeon Lee, Minsu Cho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09211.txt.

Introduction

Chunghyun Park, Seunghyeon Lee, and Minsu Cho find where on a thing a named act can be done, in full 3D. Give a shot with depth and a text ask, such as grasp the handle. Past tools mark the zone on the seen face just. The back face, hid from the shot, gets no mark, yet a robot may need it.

Their tool is called Affostruction. It first grows the full shape of the thing from the part shot. It then marks act zones on the whole shape, hid parts too. A reader in robot sight gets marks that fit a real grasp, not just a flat view. The grown shape is the key step.

Catalog record

CREATORChunghyun Park, Seunghyeon Lee, Minsu Cho
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-09211

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Affostruction: 3D Affordance Grounding with Generative Reconstruction Chunghyun Park, Seunghyeon Lee, Minsu Cho Abstract This paper addresses the problem of affordance grounding from RGBD images of an object, which aims to localize surface regions corresponding to a text query that describes an action on the object. While existing methods predict affordance regions only on visible surfaces, we propose Affostruction, a generative framework that reconstructs complete geometry from partial observations and grounds affordances on the full shape including unobserved regions. We make three core contributions: generative multi-view reconstruction via sparse voxel fusion that extrapolates unseen geometry while maintaining constant token complexity, flow-based affordance grounding that captures inherent ambiguity in affordance distributions, and affordance-driven active view selection that leverages predicted affordances for intelligent viewpoint sampling. Affostruction achieves 19.1 aIoU on affordance grounding (40.4% improvement) and 32.67 IoU for 3D reconstruction (67.7% improvement), enabling accurate affordance prediction on complete shapes. 1 Introduction Robotic manipulation requires understanding not only what kinds of objects a robot observes, but also how it can use them. In the literature, such functional properties that objects can offer —affordances—are typically predicted from complete 3D shapes 4 ; 28 ; 44 . Real-world robots, however, observe objects through RGBD cameras from limited viewpoints, resulting in partial observations with significant occlusions. Recent methods address partial observations 4 ; 11 ; 50 , but predict affordances only on visible surfaces. This limitation raises a critical issue in real-world situations: a robot viewing a mug from the front needs to reason about the occluded handle for grasping. Existing work on open-vocabulary affordance grounding from RGBD 38 ; 31 ; 30 ; 34 operates directly on partial point clouds, limiting predictions to observed surfaces, while methods for 3D reconstruction from multi-view images 10 ; 39 ; 43 ; 48 ; 42 ; 26 can complete geometry but lack mechanisms for grounding functional affordances. Figure 1 : Affostruction. Given an initial RGBD observation where functional regions for an affordance query ( e.g. , “attach a light fixture”) are poorly visible, we perform generative reconstruction to complete the 3D geometry and predict affordances on the full shape including occluded surfaces using flow-based grounding. Our affordance-driven active view selection identifies optimal viewpoints (red) that maximize visibility of high-affordance regions while avoiding ineffective views (blue), prioritizing observation of functional regions during multi-view capture. The core insight of our work is that affordance grounding on complete geometry requires first reconstructing the full shape from partial observations. Understanding where the mug handle is—and how to grasp it—requires extrapolating the occluded geometry. This motivates our approach: generating complete geometry enables affordance prediction on surfaces that were never directly observed. In this work, we introduce Affostruction , a generative framework that reconstructs complete geometry from partial observations and grounds affordances on the full shape including unobserved regions. Our approach is motivated by two key observations. First, robotic scenarios naturally provide multi-view RGBD sequences as cameras move or objects are manipulated. We leverage this through sparse voxel fusion of DINOv2 33 features that generates complete geometry by extrapolating unseen regions, aggregating information from multiple views while maintaining constant computational complexity. Second, affordances are ambiguous—multiple valid distributions exist for the same query ( e.g. , “grasp”). We address this through flow-based generation, learning distributions over affordance heatmaps rather than deterministic predictions. This probabilistic formulation naturally captures multi-modal nature of functional interactions, providing richer affordance understanding. We build upon TRELLIS 20 , a foundation model for 3D generation. TRELLIS provides a strong prior for completing 3D structures from sparse observations, trained on large-scale 3D datasets. However, it lacks the ability to process multi-view RGBD inputs and ground functional affordances, limiting its application to robotic manipulation scenarios. We extend TRELLIS with two key components: multi-view sparse voxel fusion that aggregates DINOv2 features conditioned on depth, and a flow-based affordance module that generates heatmaps from text queries. The sparse voxel representation maintains constant token complexity regardless of view count, enabling efficient aggregation. These extensions enable affordance understanding on complete geometry from multi-view RGBD images while leveraging the foundation model’s geometric priors. Our contributions are as follows: • Generative multi-view reconstruction : We propose sparse voxel fusion of DINOv2 features conditioned on depth, enabling constant-complexity generative reconstruction that extrapolates unseen geometry from partial RGBD observations, providing complete shapes for affordance prediction on occluded regions. • Flow-based affordance grounding : We introduce a flow-based generative model for affordance prediction that captures inherent ambiguity in functional interactions while conditioning on reconstructed geometry and natural language queries. • Affordance-driven active view selection : We leverage predicted affordances to guide viewpoint sampling, enabling intelligent view selection that prioritizes functional regions. 2 Related work 3D reconstruction. Multi-view 3D reconstruction methods aim to accurately recover observed geometry from multiple RGBD or RGB images. Classical approaches like TSDF fusion 2 ; 29 ; 32 ; 14 accumulate depth measurements into volumetric representations, while learning-based methods like MVSNet 48 and its variants 5 ; 49 ; 8 improve quality through cost volume matching. Recent feed-forward models like DUSt3R 43 and MASt3R 16 eliminate camera calibration requirements through learned geometric priors, achieving real-time performance crucial for robotic deployment. However, they only reconstruct observed regions without completing occluded geometry, limiting affordance grounding to visible surfaces when functional regions like handles may lie on unseen parts of objects. MCC 45 extends this direction via self-supervised cross-view consistency, but it does not handle functional or affordance-related reasoning nor effectively leverage multi-view information during inference. 3D generation. Single-view 3D generation methods 18 ; 12 ; 22 ; 21 ; 7 learn to generate plausible complete shapes from limited observations. InstantMesh 46 and LGM 40 leverage multi-view diffusion to ensure geometric consistency, while recent sparse voxel generation methods like TRELLIS 20 and XCube 36 decompose the problem into structure and appearance generation. These approaches can complete unobserved regions critical for affordance grounding on full object shapes. However, most methods operate on single RGB images, and multi-view variants 47 ; 37 ; 24 ; 17 process N views through concatenated tokens with O(N) growth, limiting inputs to 4-6 views at reduced resolution—insufficient for accurate affordance prediction in cluttered scenes. 3D affordance grounding. Recent work enables open-vocabulary grounding using vision-language models: OpenAD 31 combines 3D features with CLIP 35 embeddings, PointRefer 30 adds spatial reasoning, and Affogato 15 scales to diverse queries using Gemma3 41 -generated annotations. These methods assume complete geometry and produce deterministic predictions, limiting their ability to capture uncertainty. Some works address partial observations 4 ; 31 ; 15 , but predict affordances only on visible surfaces rather than completing occluded functional regions. We address these limitations by combining multi-view reconstruction with generative completion through depth-guided sparse voxel fusion that maintains constant O(1) token complexity regardless of view count. This enables efficient integration of dense multi-view RGBD observations while completing unobserved regions for affordance grounding. Moreover, we employ flow-based generation to capture the inherent ambiguity in affordance distributions rather than producing single deterministic outputs. Figure 2 : Affostruction overview. Our approach consists of three stages. (1) Generative multi-view reconstruction: DINOv2 33 features from multiple RGBD views are fused into sparse voxels using depth and camera parameters. A Flow Transformer conditioned on these multi-view features and trained with stochastic multi-view training extrapolates

Read the complete text