Reward Machine Inference for Robotic Manipulation

Mattijs Baert, Sam Leroux, Pieter Simoens · 2024

Read Reward Machine Inference for Robotic Manipulation by Mattijs Baert, Sam Leroux, Pieter Simoens (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10096.txt.

Introduction

Mattijs Baert, Sam Leroux, and Pieter Simoens wrote this 2024 paper on teaching robots the hidden step structure of a task from video alone. Robots can learn from demos or by trial and error. Long tasks go better when the reward is split into clear stages, but those stages are often set by hand.

The team presents a way to learn the stage machine straight from visual demos of arm tasks. Their method needs no hand set labels and no prior hints about rare reward signals. It learns the machine's states and the key events that move the robot from one state to the next. Tests on vision based tasks show the learned machine fits the task well.

Catalog record

CREATORMattijs Baert, Sam Leroux, Pieter Simoens
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reward Machine Inference for Robotic Manipulation Mattijs Baert, Sam Leroux, Pieter Simoens Abstract Learning from Demonstrations (LfD) and Reinforcement Learning (RL) have enabled robot agents to accomplish complex tasks. Reward Machines (RMs) enhance RL’s capability to train policies over extended time horizons by structuring high-level task information. In this work, we introduce a novel LfD approach for learning RMs directly from visual demonstrations of robotic manipulation tasks. Unlike previous methods, our approach requires no predefined propositions or prior knowledge of the underlying sparse reward signals. Instead, it jointly learns the RM structure and identifies key high-level events that drive transitions between RM states. We validate our method on vision-based manipulation tasks, showing that the inferred RM accurately captures task structure and enables an RL agent to effectively learn an optimal policy. IDLab, Department of Information Technology at Ghent University - imec Technologiepark 126 B-9052 Ghent, Belgium {mattijs.baert, sam.leroux, pieter.simoens}@ugent.be Introduction Combining Learning from Demonstrations (LfD) with Reinforcement Learning (RL) has empowered artificial agents to learn complex tasks from human examples in areas such as video games ( Hester et al. 2018 ) , board games ( Silver et al. 2016 ) , and robotics ( Argall et al. 2009 ; Abbeel, Coates, and Ng 2010 ) . LfD enhances RL by improving sample efficiency, which accelerates learning, and by reducing the effort required from human developers to manually specify precise task objectives. However, many LfD approaches struggle with long-duration tasks as they focus on directly learning a control policy from demonstrations ( Zhang et al. 2018 ; Mandlekar et al. 2020 ) or inferring a reward function ( Ho and Ermon 2016 ; Abbeel and Ng 2004 ; Baert et al. 2023 ) . Integrating abstract task structures, by decomposing complex tasks into manageable sub-tasks and providing structured guidance to the agent, has enabled RL to address long-horizon tasks more effectively ( Baert, Leroux, and Simoens 2024a ; Camacho et al. 2021 ) . Reward machines (RMs) ( Icarte et al. 2022 ) provide a framework for defining such abstract task structures by encoding high-level task objectives in a structured, automata-inspired format. Originally RMs were developed to extend RL for environments with non-Markovian rewards, by enhancing the agent’s state space with an abstract state layer that allow agents to retain memory of past actions. However, recent research ( Camacho et al. 2021 ) has demonstrated that RMs can also enhance performance in fully Markovian tasks by converting sparse task rewards into a denser reward signal, thereby offering more consistent guidance as the policy advances through abstract states. Despite their advantages, most current approaches require RMs to be manually specified by domain experts. In this work, we address this gap by focusing on learning reward machines directly from visual demonstrations in robotic manipulation tasks. The RM framework depends on a set of propositions that encode high-level properties of the environment and a labeling function L that assigns truth values to these propositions based on the current environment state. Unlike prior approaches, which assume these propositions or feature detectors are predefined ( Toro Icarte et al. 2019 ; Xu et al. 2021 ; Verginis et al. 2024 ; Camacho et al. 2021 ) , our approach jointly learns the RM structure and the mapping from environment states to Boolean propositions. Our method begins by capturing visual demonstrations. We leverage the observation that sub-goals are visited more frequently in expert demonstrations compared to other states ( Ghazanfari, Afghah, and Taylor 2020 ; Baert, Leroux, and Simoens 2024b ) . To identify these sub-goals, each frame is mapped to a low-dimensional feature vector. Through clustering, similar states are grouped and prototypical states (representing a certain sub-goal) can be identified. Finally, an RM is constructed capturing the sequential task structure from the demonstrations. This approach advances LfD by enabling the automatic inference of structured task representations, which can effectively guide RL agents in long-horizon tasks. We evaluate our method on a set of vision-based robotic manipulation tasks, demonstrating its effectiveness. Related Work There has been extensive research on integrating formal methods into RL. For example, several works adopt temporal logics to specify complex, long-horizon tasks ( Li, Vasile, and Belta 2017 ; Kuo, Katz, and Barbu 2020 ; Xiong et al. 2022 ; Voloshin et al. 2022 ) , while others investigate the use of automata, such as RMs, to formalize task specifications ( Araki et al. 2019 ; Araki et al. 2021 ; Icarte et al. 2022 ) . A common assumption in these approaches is that high-level knowledge, in the form of automata or temporal logic specifications, is available a priori. However, in real-world scenarios, this knowledge is often implicit and must be inferred from data. Regarding the inference of temporal logics from data, many methods focus on learning temporal logic formulas using both positive and negative examples ( Kong, Jones, and Belta 2016 ; Bombara and Belta 2021 ) . However, in the context of LfD, only positive examples (i.e., demonstrations) are typically available. Some works address learning linear temporal logic (LTL) from only positive examples ( Shah et al. 2018 ; Roy et al. 2023 ) , but these approaches usually assume that for each time step in the provided trajectories, the truth valuation of a set of Boolean propositions is known. More recent efforts aim to learn both a mapping from states to atomic propositions and the formula structure ( Baert, Leroux, and Simoens 2024b ) , although applying this in continuous state spaces remains challenging. In the domain of learning reward machines, most approaches focus on jointly learning both the RM and the policy through environment interaction. For example, discrete optimization can be used to learn an RM that decomposes the task into subproblems, such that combining their optimal memoryless policies yields an optimal solution for the original task ( Toro Icarte et al. 2019 ) . Xu et al. (2020) propose an iterative method that alternates between automaton inference, used to hypothesize RMs, and RL to optimize policies based on the current RM candidate. Inconsistencies between the hypothesis RM and observed trajectories then trigger re-learning. Verginis et al. (2024) extend this approach to settings with partially known semantics. Additionally, Xu et al. (2021) and Dohmen et al. (2022) leverage active automata inference algorithms, such as the L* algorithm, to learn RMs. Although these approaches do not rely on expert demonstrations, they assume access to the reward function of the underlying MDP as well as the labeling function that maps states to propositions. Closest to our work is the method proposed by Camacho et al. (2021) , which learns RMs for vision-based robotic manipulation tasks. However, this approach also assumes access to a predefined mapping between low-level states and high-level propositions, limiting its applicability in more general settings where such a mapping is not readily available. Background Reinforcement Learning A Markov decision process (MDP) ( Bellman 1957 ) models sequential decision-making with the following components: a state space \mathcal{S} , an action space \mathcal{A} , a discount factor \gamma\in[0,1] , a transition distribution p(s^{\prime}\mid s,a) describing the probability of reaching state s^{\prime} from state s when taking action a , an initial state distribution \mathcal{I}(s) , and a reward

Read the complete text