Object-centric Task Representation and Transfer using Diffused Orientation Fields

Cem Bilaloglu, Tobias Löw, Sylvain Calinon · 2025

Read Object-centric Task Representation and Transfer using Diffused Orientation Fields by Cem Bilaloglu, Tobias Löw, Sylvain Calinon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18563.txt.

Introduction

Cem Bilaloglu, Tobias Löw, and Sylvain Calinon move robot skills across curved objects that share no global frame. On a curved skin, words like toward or along shift sense with place, so a task taught on one shape resists transfer to the next. A flat frame cannot fix that twist.

Their tool, Diffused Orientation Fields, spreads local frames smooth across the surface. A task written in those frames then needs just a few key point matches to port to a new object. The field is computed live from raw point cloud data by a diffusion solve tied to the key points. Tests vary shape, topology, and key point noise. Smooth local frames turn a hard geometry gap into a sparse match task.

Catalog record

CREATORCem Bilaloglu, Tobias Löw, Sylvain Calinon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Object-centric Task Representation and Transfer using Diffused Orientation Fields Cem Bilaloglu, Tobias Löw, Sylvain Calinon Abstract Curved objects pose a fundamental challenge for skill transfer in robotics: unlike planar surfaces, they do not admit a global reference frame. As a result, task-relevant directions such as “toward” or “along” the surface vary with position and geometry, making object-centric tasks difficult to transfer across shapes. To address this, we introduce an approach using Diffused Orientation Fields (DOF), a smooth representation of local reference frames, for transfer learning of tasks across curved objects. By expressing manipulation tasks in these smoothly varying local frames, we reduce the problem of transferring tasks across curved objects to establishing sparse keypoint correspondences. DOF is computed online from raw point cloud data using diffusion processes governed by partial differential equations, conditioned on keypoints. We evaluate DOF under geometric, topological, and localization perturbations, and demonstrate successful transfer of tasks requiring continuous physical interaction such as inspection, slicing, and peeling across varied objects. We provide our open-source codes at our website https://github.com/idiap/diffused_fields_robotics 1 Introduction Humans routinely interact with curved objects as part of daily life, for example, slicing a banana, peeling a cucumber, or washing dishes. These tasks require continuous physical interaction with the object’s surface, often involving making and breaking contact at different regions. In such tasks, motion on (or near) the surface is guided by the pose and geometry of the object. We refer to these interactions as object-centric tasks. Unlike collision avoidance, where simplified approximations are often sufficient, object-centric manipulation that involve physical interactions demand representations that accurately capture surface geometry and exploit its structure. A core challenge in these tasks is the immense variability in object shape. Even within a single category—such as cups or bananas—geometries can differ significantly in curvature, topology, and proportions. This variability makes it infeasible to learn and store manipulation strategies for every possible shape. Although object-centric tasks are particularly challenging on complex surfaces, humans demonstrate a remarkable ability to transfer learned skills across objects with significantly different geometries. Once a skill is acquired, people can intuitively adapt their motion to new instances, even when the underlying shape changes substantially. We argue that this adaptability arises from using shape-invariant task descriptions paired with shape-aware representations that conform to the specific geometry of the object. This also directly impacts what information should be learned and transferred [ 15 ] . For example, by learning trajectories using a fixed reference frame, there is the risk of overfitting both pose and object-specific details that can act as noise and hinder generalization. Such pose and shape-dependent details should indeed not be transferred to novel objects. A straightforward approach for pose-invariant skill representation is to define motions in the object’s local reference frame [ 37 ] . More generally, encoding demonstrations from multiple reference frames enables extraction of task-relevant variability [ 2 ] . Using multiple reference frames has been extended to a dictionary of coordinate systems including cylindrical and spherical frames [ 36 ] , allowing selection of optimal geometric perspectives. Alternatively, coordinate-free descriptors [ 40 , 41 , 35 ] aim to capture invariants of motion without referring to coordinate systems. However, these pose-invariant approaches do not consider the geometry of the object and typically overlook inter-category shape variation. In contrast, keypoint-based methods [ 18 ] describe manipulation tasks using geometric costs and constraints on selected landmarks. These offer robustness to intra-category variation and surface topology changes by filtering out irrelevant geometry. Extensions such as oriented keypoints combine local reference frames with keypoints to support action specification and closed-loop policies [ 13 ] and more recent methods leveraged keypoints for visual imitation learning [ 12 , 11 ] . Neural Descriptor Fields [ 33 , 34 ] go further by encoding local task descriptors into pose-invariant neural fields, while recent work leverages foundation models for learning task descriptors [ 42 ] . Instead, policy learning approaches learn perception-to-action mapping from sensory inputs such as RGB [ 4 ] or point clouds [ 44 ] . Recent reactive-compliance variants explicitly integrate tactile/force feedback to stabilize contact [ 14 , 43 ] . However, these approaches are fundamentally data-driven—requiring demonstrations and offline training—and aim to learn end-to-end policies rather than a modular, geometry-aware representation that can be reused across objects, controllers and tasks. As an alternative to learning-based approaches, geometric methods use discrete differential geometry for task transfer across surfaces. Functional mapping methods [ 24 , 8 , 7 ] enable transferring functions over near-isometric surfaces given sparse correspondences; however, they are limited to open-loop position trajectories constrained to object surfaces. Other approaches use signed distances and surface parameterizations for ultrasound scanning [ 9 ] , approximate exponential maps for learning from demonstration on meshes [ 39 ] , and logarithmic maps for interactive grasping design [ 16 ] . While effective within their scopes, these methods generally assume clean meshes, making them less suited to noisy, partial point clouds collected at runtime. Moreover, they do not provide a controller-agnostic representation that can be integrated as a module within other planning, learning, or control frameworks. Other efforts targeting continuous physical interaction on curved surfaces address specific problems (e.g., bimanual coordination [ 38 ] or force generation [ 1 ] ) but do not provide fine-grained adaptation to novel object instances. A key insight is that object-centric tasks like cleaning or slicing on a planar surface can often be reduced to simple, repetitive motions—such as up/down or back/forth—thanks to the presence of a consistent global frame. In contrast, curved or irregular objects lack such a global reference frame, making task specification and transfer significantly more challenging. Oriented keypoints [ 13 ] or using multiple body-fixed reference frames [ 2 ] can provide geometric structure for local interactions, however they are insufficient for continuous interactions with breaking and making contact near the surface. For that purpose, we construct a smoothly varying continuous representation of local frames throughout the robot’s workspace conditioned on surface geometry and a few semantic keypoints. These local frames act as a geometric scaffold for expressing manipulation actions. For instance, a slicing or probing task might be described using a pose- and shape-invariant task description “slide along the object, go down and go up” : an instruction that remains well-defined across object instances, categories, and poses. Therefore, our approach to represent and transfer object-centric tasks consists of (i) Diffused Orientation Fields (DOF) representing a smooth field of local reference frames conditioned on the object geometry and (ii) shape-invariant local actions defined in those frames. These local actions can be generated by various high-level controllers ranging from human in teleoperation to trajectory optimizer in planning and a policy programmed using primitives or learned using reinforcement learning. Our method reduces task-transfer to a novel object to computing its DOF from online vision and depth input while reusing the same shape-invariant local actions. This requires the representation to encode task-relevant geometric information across the workspace, support online computation and tolerate noisy and partial inputs. To construct DOF online, we draw inspiration from computer graphics. We use a partial differential equation (PDE) called the diffusion equation (also known as the heat equation), which provides a principled way to extend scalar

Read the complete text