Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks

Amin Berjaoui Tahmaz, Ravi Prakash, Jens Kober · 2025

Read Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks by Amin Berjaoui Tahmaz, Ravi Prakash, Jens Kober (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19607.txt.

Introduction

Amin Berjaoui Tahmaz, Ravi Prakash, and Jens Kober (2025) present a learning framework that teaches robots to handle contact tasks in sequence. Jobs such as lifting a block, opening a door, pushing an object, and wiping a surface each need a different touch: stiff for force, soft for safety. The framework stacks simple behavior units in a hierarchy and lets the robot vary the stiffness of its arm while each unit runs.

Three parts make it work: an action space that includes stiffness choices, a controller that adjusts stiffness during each motion, and a coupling method that guides exploration while favoring gentle contact. In training and tests, the system learned faster, reused its units in new combinations, and succeeded more often than leading methods. Trials on a real robot confirmed that skills learned in simulation carried over, a step toward machines that adapt their touch to the task.

Catalog record

CREATORAmin Berjaoui Tahmaz, Ravi Prakash, Jens Kober
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks Amin Berjaoui Tahmaz, Ravi Prakash, Jens Kober Abstract This paper presents an Impedance Primitive-augmented hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework’s sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks. I Introduction Realistic manipulation tasks involve a prolonged sequence of motor skills in varying environments. For decades, the challenge of enabling robotic manipulators to solve realistic long-horizon tasks has persisted. While existing research has made strides in addressing important aspects of long-horizon tasks, a critical gap remains in the context of contact-rich environments, highlighting a crucial area that requires further exploration and development. An example can be found in a common manipulation task: object sorting. A robot should be able to plan a series of precise actions over time while adjusting its positioning and applied forces to accommodate objects of varying shapes and sizes while also taking the interaction environment into consideration. This paper focuses on the intersection of deep reinforcement learning (DRL) and adaptive stiffness control to address this longstanding challenge. Prior works have extensively explored robotic manipulation in long-horizon applications. Conventional methods often use state machines [ 1 ] [ 2 ] or symbolic reasoning [ 3 ] [ 4 ] to learn action sequences for solving a task. However, these approaches explicitly design the decision-making sequence, which may introduce constraints that limit adaptability to different tasks and contribute to error accumulation throughout the task sequence. In response to these limitations, learning techniques such as hierarchical reinforcement learning (HRL) [ 5 ] have been employed, establishing themselves as a common approach for problems requiring sequential decision-making. When deploying long-horizon frameworks in contact-rich environments, the integration of stiffness control becomes crucial for adapting to external forces and uncertainties during task execution. This adaptability ensures precision and stability in navigating contact-rich environments. However, despite a substantial body of research dedicated to variable stiffness control, current approaches are primarily tailored to short-horizon applications. These methods typically involve designing controllers that adjust end-point force in response to environmental forces [ 6 ] , adapting impedance and damping parameters through learning techniques [ 7 ] [ 8 ] , and learning from a human demonstrator [ 9 ] [ 10 ] . Fig. 1: Figure shows the augmentation of the impedance primitive into HRL policy. This paper aims to bridge the gap between sequential task planning and adaptive stiffness control using a DRL framework. We design an HRL framework, as shown in Figure 1 , that selects a high-level action primitive from a pre-defined library and outputs an initial estimate for controller parameters for low-level control. During primitive execution, an adaptive controller is initiated to optimize the robot’s stiffness, aiming for an balance between safety (reducing interaction forces with the environment) and performance (ensuring task completion). This design allows the robot to dynamically optimize stiffness parameters, enabling it to transition between high stiffness for precision tasks and increased compliance for enhanced adaptability. We present experiments conducted in both simulation and the real world, focusing on sequential tasks that deal with different contact challenges. Our results highlight notable advantages when compared to a state-of-the-art baseline. The remainder of this paper is structured as follows: Section II discusses related work. Section III defines the problem statement. Section IV provides necessary background and preliminaries. Our proposed Impedance Primitive-augmented HRL (IMP-HRL) approach is introduced in Section V . Section VI presents experimental results and analysis. Section VII concludes the paper with a summary of findings and future research directions. II Related Works Sequential Planning: Extensive work in task and motion planning (TAMP) spans various robotics applications, involving explicit decision-making frameworks and machine learning for learned behavior sequences. Common approaches employ hierarchical task planning, combining high-level planners with low-level controllers. In robot manipulation, this often takes the form of finite state machines [ 1 , 11 , 12 ] or behavior trees [ 13 , 14 ] as high-level controllers. Similar methods use symbolic reasoning [ 15 , 16 , 17 ] , representing high-level tasks and constraints with symbols. Although these methods offer explainability, their pre-defined nature limits adaptability to real-world variability, leading to suboptimal performance. Our proposed framework addresses this by learning the high-level planner and optimizing low-level controller parameters for better generalization and robustness. Recently, learning approaches have emerged to overcome these limitations. Imitation learning (IL) is a key candidate for sequential planning, enabling robots to learn demonstrated behavior sequences. Behavior cloning, a well-established IL method, has robots replicate demonstration sequences [ 18 , 19 , 20 ] , but this limits generalizability. Advanced IL methods aim to generalize learned sequences [ 21 , 22 , 23 ] , yet they still struggle to adapt to new environments. Our framework adapts action sequences to the environment state, addressing this limitation and mitigating suboptimal performance from human error in demonstration data. Hierarchical Reinforcement Learning (HRL) has gained attention for long-horizon planning. State-of-the-art approaches like MAPLE [ 24 ] , RAPS [ 25 ] , and STAP [ 26 ] train hierarchical policies to choose and execute primitives from a behavior library. Despite handling complex tasks and improving sample efficiency, these methods rely on static controllers, which hinder performance in contact tasks and pose risks in real-world settings. Our method builds on these concepts, optimizing stiffness to maximize compliance without compromising task success. Variable Stiffness Control: Existing methods for adapting the stiffness of an impedance controller typically use task-specific impedance profiles. Common approaches include learning from demonstration methods, such as Dynamic Motion Primitives [ 27 , 28 , 29 ] or Gaussian Mixture Models [ 9 , 30 ] . Alternatively, some methods schedule variable stiffness gains for different task phases [ 31 , 32 , 33 ] . Despite their ease of application, these methods struggle to generalize stiffness profiles across tasks and depend on expert demonstrators. RL has emerged as a promising method for learning stiffness profiles. Some methods bootstrap the RL policy with initial stiffness demonstrations [ 34 , 35 , 36 ] to accelerate learning, which are then optimized for specific tasks. However, the reliance on expert demonstrators remains an issue. Other RL approaches focus on designing an appropriate action space in which an agent samples impedance parameters as actions to adapt controller behavior. For adaptive stiffness applications, an impedance action space allows the agent to learn stiffness and damping parameters in joint space [ 37 ] and end-effector space [ 8 ] . Similar approaches use residual reinforcement learning, where a policy outputs

Read the complete text