Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation

Tzu-Jung Lin, Jia-Fong Yeh, Hung-Ting Su, Chung-Yi Lin, Yi-Ting Chen, Winston H. Hsu · 2025

Read Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation by Tzu-Jung Lin, Jia-Fong Yeh, Hung-Ting Su, Chung-Yi Lin, Yi-Ting Chen, Winston H. Hsu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06240.txt.

Introduction

Tzu-Jung Lin and co-authors ask where a mobile robot should park itself before trying to manipulate an object. In open-vocabulary mobile manipulation, the robot must handle items described in plain words. Picking a base position by closeness alone ignores whether the spot actually affords a good grip, causing frequent failures.

Their zero-shot framework pairs vision-language understanding with geometric checks in a coarse-to-fine search. Special representations blend affordance cues with an obstacle map, so reasoning reaches beyond the camera view. Coarse semantic hints steer the search first, then geometry refines the final spot. Across five varied tasks the system reached an eighty-five percent success rate, well ahead of classical planners.

Catalog record

CREATORTzu-Jung Lin, Jia-Fong Yeh, Hung-Ting Su, Chung-Yi Lin, Yi-Ting Chen, Winston H. Hsu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06240

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation Tzu-Jung Lin, Jia-Fong Yeh, Hung-Ting Su, Chung-Yi Lin, Yi-Ting Chen, Winston H. Hsu Abstract In open-vocabulary mobile manipulation (OVMM), task success often hinges on the selection of an appropriate base placement for the robot. Existing approaches typically navigate to proximity-based regions without considering affordances, resulting in frequent manipulation failures. We propose Affordance-Guided Coarse-to-Fine Exploration, a zero-shot framework for base placement that integrates semantic understanding from vision-language models (VLMs) with geometric feasibility through an iterative optimization process. Our method constructs cross-modal representations, namely Affordance RGB and Obstacle Map+ , to align semantics with spatial context. This enables reasoning that extends beyond the egocentric limitations of RGB perception. To ensure interaction is guided by task-relevant affordances, we leverage coarse semantic priors from VLMs to guide the search toward task-relevant regions and refine placements with geometric constraints, thereby reducing the risk of convergence to local optima. Evaluated on five diverse open-vocabulary mobile manipulation tasks, our system achieves an 85% success rate, significantly outperforming classical geometric planners and VLM-based methods. This demonstrates the promise of affordance-aware and multimodal reasoning for generalizable, instruction-conditioned planning in OVMM. Figure 1: Examples of failure cases caused by base placements without affordance awareness. Left: The robot cannot open the cabinet because it is not facing the drawer. Middle: The robot cannot grasp the pot handle due to misalignment. Right: The robot fails to place the mug on the shelf as it is not facing the open side. These failures arise from a lack of joint reasoning over task intent and geometric feasibility, leading to semantically misaligned placements that prevent successful manipulation. 1 Introduction In open-vocabulary mobile manipulation (OVMM), selecting an appropriate base placement is critical for successful task execution. However, prior navigation systems ( 26 ; 5 ; 17 ; 23 ) often treat the task as complete once the robot reaches a location near the target. In practice, mere proximity does not guarantee effective manipulation. Specifically, determining the base placement presents two key challenges. First , the robot must reason jointly about geometric feasibility and semantic intent . It needs to identify a collision-free location that maintains appropriate distance and aligns with task-relevant affordances. For example, to open a cabinet, the robot must position itself in front of the cabinet with enough clearance for effective interaction. Second , the robot must reason globally despite limited perceptual input . A narrow field of view restricts spatial awareness and may cause the robot to overlook suitable placements. For instance, if the area in front of the cabinet is not visible in the RGB image, the robot cannot evaluate it as a potential base location. Existing approaches struggle to address the aforementioned challenges. In many OVMM ( 5 ; 17 ) or navigation ( 22 ; 6 ) works, once the target object’s location is known, classical path planners such as A* ( 3 ) and RRT* ( 8 ) are used to navigate the robot close to the object. These planners rely solely on geometric heuristics and lack task-level semantic understanding. As a result, they often produce placements that fail to account for task affordances, as illustrated in Figure 1 . Conversely, semantic methods that leverage vision-language models (VLMs) can infer high-level task intent ( 14 ; 20 ) , but typically overlook geometric feasibility and reachability constraints. Furthermore, these methods often rely on a single RGB image, which limits their ability to reason about occluded or unseen areas. In this paper, we propose a novel zero-shot framework, Affordance-Guided Coarse-to-Fine Exploration , which integrates both semantic reasoning and geometric understanding for effective base placement, especially under limited field of view. Our approach introduces two key innovations: (1) Cross-modal representations , specifically Affordance RGB and Obstacle Map+ , which align task intent with spatial layout while mitigating perceptual limitations caused by occluded or unseen regions. (2) A coarse-to-fine optimization process that begins with sampling in semantically guided regions and iteratively refines the search toward geometrically feasible placements. This enables the robot to satisfy both semantic and geometric requirements. Our system requires no task-specific supervision and operates solely on natural language instructions, RGB-D images, and an obstacle map. It generalizes across a wide range of OVMM tasks with varying spatial and semantic demands. Experiments show that our method achieves an overall success rate of 85%, significantly outperforming classical and semantic-only baselines. These results underscore the effectiveness of our key component, i.e. affordance-aware and multimodal reasoning, in enabling instruction-driven base placement for open-vocabulary mobile manipulation. Our main contributions are: • We identify base placement as a critical challenge in OVMM, driven by the need to reason about both semantics and geometry under limited perceptual input. • We propose a coarse-to-fine strategy that unifies semantic and geometric cues via Affordance RGB and Obstacle Map+ , addressing limited field of view and overcoming the limitations of prior single-focus methods. • We develop a generalizable zero-shot system that achieves 85% success across tasks, outperforming prior classical and semantic-only approaches. 2 Related Work Open-Vocabulary Navigation and Manipulation Recent vision-language models (VLMs) ( 15 ; 24 ; 12 ) have shown strong potential in open-vocabulary navigation and manipulation. Some systems build semantic maps using VLM features to enable language-guided planning. For example, FindAnything ( 10 ) constructs object-centric submaps, while CLIP-Fields ( 22 ) and VLMaps ( 6 ) embed vision-language features into 3D environments. CLIP-Fields ( 22 ) also projects weakly-supervised features to generate semantic maps. Methods like USA-Nets ( 1 ) and GOAT ( 2 ) locate targets using language-image matching, then guide navigation using geometric planners like A*. These techniques extend to open-vocabulary mobile manipulation (OVMM). OK-Robot ( 5 ) builds a VoxelMap from RGB-D scans and uses CLIP ( 18 ) , SAM ( 9 ) , and OWL-ViT ( 13 ) for grounding and segmentation. It then queries VLMs to plan collision-free paths. COME-robot ( 28 ) incorporates GPT-4V for task planning and failure recovery. Our work differs by focusing specifically on selecting semantically meaningful and physically feasible base placements. Visual Prompting for Robotics Visual prompting is an emerging tool in robotic VLMs. While Set-of-Mark Prompting (SoM) ( 25 ) was originally developed for visual grounding tasks, its underlying techniques have been extended to symbolic prompting in systems like PIVOT ( 14 ) , which frames spatial tasks as iterative visual question answering and refines VLM predictions through repeated annotation and selection. CoPa ( 4 ) , ReKep ( 7 ) , and MOKA ( 27 ) use visual prompts to infer keypoints or constraints for manipulation. KAGI ( 11 ) utilizes keypoints to define dense rewards in reinforcement learning, while CoNVOI ( 20 ) applies region-level prompting for navigation. However, these methods rely on single-view RGB inputs, limiting their ability to reason about occluded or unseen regions. In contrast, our approach applies visual prompting directly to obstacle maps, enabling global inference beyond the current view. Recent Advances in Base Placement Recent work addresses base placement by integrating perception and planning. MoMa-Pos ( 21 ) optimizes base placement by modeling task-relevant articulated objects using frontal-view images, but requires object-specific modeling and lacks generalization to novel categories. MoMa-Kitchen ( 27 ) introduces a large

Read the complete text