Grounded Task Axes: Zero-Shot Semantic Skill Generalization via Task-Axis Controllers and Visual Foundation Models

M. Yunus Seker, Shobhit Aggarwal, Oliver Kroemer · 2025

Read Grounded Task Axes: Zero-Shot Semantic Skill Generalization via Task-Axis Controllers and Visual Foundation Models by M. Yunus Seker, Shobhit Aggarwal, Oliver Kroemer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11680.txt.

Introduction

Yunus Seker, Shobhit Aggarwal, and Oliver Kroemer present a 2025 robot method that transfers a skill to a new object with no extra training. The first author's middle initial appears in the byline. Moving skills between objects is a core problem in open world robot work. Objects differ in shape at a high level, yet the low level contact control stays much the same. Their method splits a skill into a ranked list of controllers. Each controller acts along one axis.

Each controller is anchored to key points and axes on the object, such as a screw head and its shaft. A vision foundation model then finds matching points on a new object by meaning, which grounds the same skill there. Real robot tests covered screwing, pouring, and scraping with a spatula. The authors report robust, flexible transfer across all three tasks.

Catalog record

CREATORM. Yunus Seker, Shobhit Aggarwal, Oliver Kroemer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11680

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grounded Task Axes: Zero-Shot Semantic Skill Generalization via Task-Axis Controllers and Visual Foundation Models M. Yunus Seker, Shobhit Aggarwal, Oliver Kroemer Abstract Transferring skills between different objects remains one of the core challenges of open-world robot manipulation. Generalization needs to take into account the high-level structural differences between distinct objects while still maintaining similar low-level interaction control. In this paper, we propose an example-based zero-shot approach to skill transfer. Rather than treating skills as atomic, we decompose skills into a prioritized list of grounded task-axis (GTA) controllers. Each GTAC defines an adaptable controller, such as a position or force controller, along an axis. Importantly, the GTACs are grounded in object key points and axes, e.g., the relative position of a screw head or the axis of its shaft. Zero-shot transfer is thus achieved by finding semantically-similar grounding features on novel target objects. We achieve this example-based grounding of the skills through the use of foundation models, such as SD-DINO, that can detect semantically similar keypoints of objects. We evaluate our framework on real-robot experiments, including screwing, pouring, and spatula scraping tasks, and demonstrate robust and versatile controller transfer for each. I Introduction Transferring robot manipulation skills across diverse objects remains a fundamental challenge in open-world robotics. Humans intuitively generalize tasks across varied objects by identifying shared semantic and functional characteristics, even when confronted with substantial differences in appearance or geometry. Robotic systems, however, typically lack this intuitive adaptability, often failing to generalize when faced with even minor variations. Addressing this limitation requires robotic systems capable of abstracting tasks at a semantic level while still reliably performing low-level physical interactions. A promising strategy to address this issue involves decomposing complex manipulation skills into modular components defined relative to object-centric coordinate frames or task-specific axes. Early work introduced structured representations of robot manipulation tasks based on meaningful task frames and axes [ 1 , 2 , 3 ] . Building upon these concepts, Sharma et al. [ 4 , 5 ] further advanced this approach by defining tasks as prioritized sequences of modular controllers. These modular frameworks significantly improved robustness, interpretability, and flexibility in robot manipulation. However, despite their effectiveness, existing task-axis methods still generally require extensive task-specific training, heavily annotated datasets or multiple demonstrations, and predefined sets of controllers, thus limiting their ability to generalize skills in zero-shot. To overcome these limitations, we propose a novel zero-shot approach to semantic grounding of task-axis controllers. Our framework significantly expands the generalization capabilities of task-axis controllers by leveraging recent advancements in visual foundation models. Specifically, we propose a divide-and-conquer methodology that decomposes complex manipulation tasks into individual grounded task-axis (GTA) controllers. Each GTA controller specifies the low-level robot action—such as force application or positional adjustment—along clearly defined axes and keypoints grounded to the objects’ local geometry. Rather than relying on extensive retraining or demonstrations, we enable zero-shot generalization through example-based semantic grounding using powerful visual foundation models like SD-DINO [ 6 ] . Visual foundation models, such as SD-DINO, integrate semantic and geometric information, learned from large-scale pretraining, allowing for robust identification of semantically similar keypoints across highly varied object instances. In our framework, we exploit these capabilities to perform keypoint matching between reference and novel objects purely from RGB-D images. This matching facilitates accurate grounding of task-axis controllers in novel scenarios without task-specific demonstrations or object-specific training. Such a semantic and visual approach contrasts sharply with traditional geometric-based methods, offering enhanced robustness to significant variations in object shapes, textures, colors, and viewing angles. Our method makes four key contributions to skill generalization in robotic manipulation: Modular Skill Representation: We represent complex skills as prioritized sets of grounded task-axis controllers, improving interpretability,providing task-tailored manipulation control, and enabling sub-skill controller reuse. Semantic Grounding with Visual Foundation Models: We use SD-DINO to match keypoints across object instances, allowing accurate grounding in novel scenarios. Zero-Shot Generalization: Our framework generalizes skills to novel objects without any training or demonstrations. Real-World Evaluation: We validate our approach on diverse tasks, pan scraping, pouring, and screwing, demonstrating zero-shot execution across varied object geometries. Fig. 2: Vision pipeline for grounding task-axes: Given a reference object annotated with keypoints and a target object, GTA uses SD-DINO as a vision backbone to extract visual features. We perform keypoint matching to find corresponding keypoints on the target object. Using these keypoints and 3D point cloud data, we ground the object-centered task-axis controllers in real-world. II Related Work Task-Axis Controllers: Task-axis frames have long been used in robotic manipulation to define motions relative to meaningful reference frames. Early works [ 1 , 2 , 3 ] introduced constraint-based formulations for specifying manipulation along task-axes. Later methods further developed this idea, defining task-specific motions using fixed task-frames or axes for reliable execution [ 7 , 8 , 9 , 10 ] . More recently, task-axis controllers have been used as modular components for structuring complex behaviors. Sharma et al. [ 4 ] proposed composing such controllers via reinforcement learning, and later extended the framework to infer controller parameters from visual input via dense correspondence learning [ 5 ] . However, these methods depend on task-specific training. In contrast, our method performs inference in a zero-shot manner without retraining. Imitation Learning and Trajectory Generation: Skill learning often involves training trajectory generators using imitation learning. Methods like DMPs [ 11 , 12 , 13 ] often require endpoints to be specified and a suitable coordinate frame to be selected for generalization. Rather than identifying a task-oriented coordinate frame, recent diffusion-based policies [ 14 , 15 ] , generate trajectories in global frames. Perceiver-Actor [ 15 ] directly predicts a target gripper frame and uses a motion planner for execution. By mapping scenes to compact keypoints and axes, our method can reduce input dimensionality and improve data efficiency. In the future, we plan to incorporate imitation learning approaches for trajectory generation into our controller library. Visual Foundation Models for Correspondences: Vision foundation models have shown strong performance in establishing semantic and dense correspondences. Models like CLIP [ 16 ] , DiFT [ 17 ] , and DINO-ViT [ 18 ] learn semantic features from large-scale pretraining and enable pixel-level matching without supervision [ 19 ] . We leverage SD-DINO [ 6 ] to compute sparse semantic correspondences for robust keypoint matching. Our system remains modular and can incorporate future vision models seamlessly. Keypoint Correspondences for Manipulation: Recent works like DINOBot [ 20 ] , SKIL [ 21 ] , and MAGIC [ 22 ] extract keypoints from pretrained vision models to enable generalization across object categories. However, these methods train task-specific policies from demonstrations or rely on real-to-sim transfer for planning [ 23 , 24 , 25 ] . By contrast, our method operates entirely in a zero-shot setting, without demonstrations or policy training. III Method III-A Task-Axis Controllers Our proposed modular framework creates skills by combining multiple grounded task-axis controllers in parallel. This skill structure builds on the work of Sharma and Liang et al [ 4 , 5 ] , and we provide a more general formulation for these types of skills. It should be noted that our skills also build upon more classical task-frame formulations of robot manipulation [ 1 , 2 , 3 ] ,

Read the complete text