Hierarchical meta-reinforcement learning via automated macro-action discovery Authors: Minjae Cho, Chuangchuang Sun Published: 2026 ABSTRACT Meta-Reinforcement Learning (Meta-RL) enables fast adaptation to new testing tasks. Despite recent advancements, it is still challenging to learn performant policies across multiple complex and high-dimensional tasks. To address this, we propose a novel architecture with three hierarchical levels for (1) learning task representations, (2) discovering task-agnostic macro-actions in an automated manner, and (3) learning primitive actions. The macro-action can guide the low-level primitive policy learning to more efficiently transition to goal states. This can address the issue that the policy may forget previously learned behavior while learning new, conflicting tasks. Moreover, the task-agnostic nature of the macro-actions is enabled by removing task-specific components from the state space. Hence, this makes them amenable to re-composition across different tasks and leads to promising fast adaptation to new tasks. Also, the prospective instability from the tri-level hierarchies is effectively mitigated by our innovative, independently tailored training schemes. Experiments in the MetaWorld framework demonstrate the improved sample efficiency and success rate of our approach compared to previous state-of-the-art methods. The code that was used in the current study is available in https://github.com/Mgineer117/himeta. Introduction Reinforcement Learning (RL) has rapidly advanced, enhancing decision-making capabilities in high-dimensional and complex systems. Recent innovations have expanded RL to address diverse challenges, including multi-task RL, safe RL, and offline RL, thus broadening its practical applications. Among these advancements, meta-learning emphasizes training an agent to adapt to new tasks by leveraging previously acquired knowledge. This often involves balancing the performance across multiple training tasks, exposing the learner to diverse task-parameters to facilitate future adaptation to any prospective test task within the same distribution. Hence, due to the inherent challenge in meta-learning1, common methodologies are typically tested on narrow task distributional learning scenarios such as a jumping robot with varying loads (altering transition dynamics) and a running agent with different velocity criteria (changing optimality). Although it faces challenges when extended to more complex meta-learning scenarios, task representation learning2 shows its potential in such environments by identifying shared structures across given training tasks. The discovered representations offer fundamental insights into tasks, aiding the decision-maker in adapting and performing effectively on new tasks. Building on task representations, which expand the input space with additional conditioners, we aim to further distribute the policy’s information load by utilizing high-level actions, or macro-actions. Specifically, macro-action provides directional action vectors to the low-level policy. Then, the policy fills with detailed low-level control input along the given direction. This approach is particularly useful for learning multiple tasks and adapting to new ones in hard-parameter sharing. In this setting, effective distribution of information processing can handle challenging task-specific instructions, ensuring that the policy retains its previously acquired optimal behavior in a particular task while learning new tasks. The illustration is given in Fig. 1.Fig. 1This overview outlines our approach. Our higher-level architecture, HiMeta, offers high-level directional action predictions to the policy, on which the policy generates primitive actions based on these directions. Additionally, the directional action prediction is trained using a modified VAE designed to link the current state to the desired goal state. This enables the effective distribution of decision-making tasks, facilitating solutions for complex, high-dimensional multi-task and meta-learning scenarios. This overview outlines our approach. Our higher-level architecture, HiMeta, offers high-level directional action predictions to the policy, on which the policy generates primitive actions based on these directions. Additionally, the directional action prediction is trained using a modified VAE designed to link the current state to the desired goal state. This enables the effective distribution of decision-making tasks, facilitating solutions for complex, high-dimensional multi-task and meta-learning scenarios. In short, beyond the existing bi-level representation learning (task-representation | low-level control), we introduce a tri-level method (task-representation | high-level control | low-level control) that includes an additional layer for constructing macro-actions. For task-representation, we adopt the goal state concept from hierarchical RL literature3,4. For high-level control inputs, we propose a modified Variational AutoEncoder (VAE) to identify an effective placeholder for low-level control. This placeholder provides both sign-based and magnitude-level guidance, enabling the policy to refine it into precise low-level controls. In the architecture, the encoder predicts macro-actions, while the decoder uses imputations to bridge the current state with the goal state. Therefore, this approach brings appealing properties in complex, sparse, long-horizon problems. We provide mathematical views of the mechanism of the encoder and decoder. For stacking three levels of layers, we design each to have an independent role in decomposition to avoid the “curse of hierarchy”5,6, which refers to the instability of learning higher-level layers caused by frequent dynamical transitions by the low-level decision-maker. While this challenge has previously limited hierarchical learning to bi-level layers, our novel solution of independent training schemes tailored for each hierarchical layer effectively enables three-layer hierarchical learning, demonstrating its effectiveness in broad meta-task learning. Our contributions are as follows:Proposing a novel architecture to automatically discover macro-actions on which the policy fills the low-level input. This additionally prevents the loss of optimality in a particular task while learning new tasks by the distribution of information load.Introducing a training scheme and techniques to avoid the “curse of hierarchy,” enabling multi-level layers and collaborative learning across multiple tasks.Conducting extensive meta-learning experiments on complex and diverse task distributions, achieving state-of-the-art performance compared to baselines.This letter outlines the preliminaries of meta-learning, offering insights into hierarchical decomposition in the following section “Preliminaries”. Additionally, we detail our algorithmic approach and highlight the key insights that drive the state-of-the-art performance of our method in “HiMeta: hierarchical meta-reinforcement learning”, accompanied by extensive experiments in “Experiments”. Finally, we provide our concluding remarks and discuss potential improvements and future work in “Conclusions”. Proposing a novel architecture to automatically discover macro-actions on which the policy fills the low-level input. This additionally prevents the loss of optimality in a particular task while learning new tasks by the distribution of information load. Proposing a novel architecture to automatically discover macro-actions on which the policy fills the low-level input. This additionally prevents the loss of optimality in a particular task while learning new tasks by the distribution of information load. Introducing a training scheme and techniques to avoid the “curse of hierarchy,” enabling multi-level layers and collaborative learning across multiple tasks. Introducing a training scheme and techniques to avoid the “curse of hierarchy,” enabling multi-level layers and collaborative learning across multiple tasks. Conducting extensive meta-learning experiments on complex and diverse task distributions, achieving state-of-the-art performance compared to baselines. Conducting extensive meta-learning experiments on complex and diverse task distributions, achieving state-of-the-art performance compared to baselines. Preliminaries Reinforcement learning operates within the mathematical framework of Markov Decision Processes (MDP)7, characterized by the tuple \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle \mathcal {S}, \mathcal {A}, T(\cdot ), R(\cdot ), \gamma \rangle$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S} \in \mathbb {R}^s$$\end{document} represents the state set (observation), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {A} \in \mathbb {R}^a$$\end{document} is the action set, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T(\cdot ): \mathcal {S} \times \mathcal {A} \rightarrow \mathcal {S}$$\end{document} denotes the transition function, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R(\cdot ): \mathcal {S} \times \mathcal {A} \rightarrow \mathbb {R}$$\end{document} is a reward function with γ as a discount factor. RL for any arbitrary task, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {T}$$\end{document}, comprises its own transitional and reward dynamics. Formally, the definition of a task, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {T}$$\end{document}, is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\langle T(\cdot ), R(\cdot ) \rangle \in \mathcal {T}$$\end{document}. The goal of deep RL is to find a policy parameterized by θ, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi _\theta (a|s)$$\end{document}, that maps the state to an optimal action, yielding the highest cumulative rewards (optimal trajectory) under task \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {T}$$\end{document}, as shown below:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \max _{\theta } \mathcal {J}(\pi _\theta , \mathcal {T}) = \mathbb {E}_{\begin{array}{c} a \sim \pi _\theta (s),\\ s' \sim T(s,a) \end{array}} \left[ \sum _{t=0}^T \gamma ^t R(s_t, a_t) \right] \end{aligned}$$\end{document} Meta-Learning Meta-learning extends multi-task learning by aiming for rapid adaptation to unseen tasks using prior knowledge. Formally, the trained policy aims to maximize the prospective optimality objective of the test task \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {T}_{\text {test}}$$\end{document}:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \max _\theta \mathcal {J}(\pi _\theta , \mathcal {T}_{\text {test}}), \quad \text {where} \quad \theta = \arg\max_\theta \mathcal {J}(\pi _\theta , \mathcal {T}_{train}) \end{aligned} \end{aligned}$$\end{document} Meta-training is typically achieved by conditioning task representations or finding a parametric space that is close to the new task’s optimal space as above. Task representation aims to discover common or distinct knowledge across tasks to better analyze the test tasks. This is typically done by posterior sampling over the variable of interest x, a latent task representation, using a variational autoencoder with the following objective:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \log p(x) = \mathbb {E}_{q(z|x)}[\log p(x|z)] - \text {KL}(q(z|x) \Vert p(z)) \end{aligned}$$\end{document}where p(· ) denotes the probability, q(· ) represents the posterior distribution, and KL is Kullback-Leibler divergence. The representation is a type of informative label, and it is then conditioned to the policy, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi _\theta$$\end{document}, to endow better differentiation. This creates useful input spaces that are easily shared across multiple tasks, inducing positive knowledge sharing. The construction of posterior distribution of representations has proven effective in many previous studies involving complex meta-adaptation tasks, which are relevant to our approach. Details of relevant methods are described in Related Works. Hierarchical Reinforcement Learning Hierarchical RL is introduced as a solution for tasks with long horizons and sparse rewards by decomposing the original problem into several sub-problems. This is typically achieved with a new optimality objective, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mathcal {J}}$$\end{document}, as shown below by additionally conditioning the policy on a goal z:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\mathcal {J}} = \mathbb {E}_{\begin{array}{c} a \sim \pi _\theta (s,z), \\ s' \sim T(s,a) \end{array} }\left[ \sum _{t=0}^T \gamma ^t R(s_t, a_t) + \alpha \mathbb {I}(s_t = z) \right] \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {I}(\cdot )$$\end{document} is an indicator function that adds a reward bonus, scaled by α, when the policy successfully reaches the given goal z. The z can be a latent representation, goal, or a raw state, goal state. With this definition, the policy is efficiently guided in sparse rewards and long-horizon settings4,5,8,9. HiMeta: Hierarchical Meta-Reinforcement Learning A high information load on the decision-maker can impede learning by overwhelming its capacity to process and integrate information. This issue becomes especially apparent in multi-task settings, where improving performance on one task can inadvertently degrade performance on another. To address this, we propose decomposing task representations into a more fundamental, task-agnostic language that enables an action-guided learning framework. We introduce three handcrafted definitions of goal states and incorporate an additional layer from prior bi-level approaches that identifies future action embeddings capable of connecting the current state to these goal states. In essence, our algorithm employs carefully designed architectures and loss functions to discover macro-actions by bridging the action and state spaces through a VAE and imputation mechanisms. We introduce a tri-level hierarchical learning framework in which each layer serves as an independent module for task representation (high level), macro-action discovery (intermediate level), and policy learning (low level). These modules are updated sequentially, as illustrated in Algorithm 1, with the high- and intermediate-level updates performed first and the low-level update following at the same frequency. We provide the implementation detail in Table 3.Fig. 2Our algorithm consists of three hierarchical layers: high (task representation learning), intermediate (macro-action discovery), and low (primitive actions discovery). The high-level layer discovers a task representation y, using a recurrent unit, simultaneously with value-function training. The intermediate layer analyzes the representation given the current state to determine the macro-action z which is the (+/-) sign of actions. This is trained via VAE using the missing information technique. The state element \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego}$$\end{document} is a subset of the state set, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}^{ego} \in \mathcal {S}$$\end{document} that includes everything except the agent’s self-state. The subsequent loss in decoder with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego}$$\end{document} will shape macro-actions, z, task-agnostic and compact. The representation and macro-action are then conditioned in the low-level policy to make primitive decisions. Gradients do not overflow between hierarchical layers, ensuring each layer’s independent role. Our algorithm consists of three hierarchical layers: high (task representation learning), intermediate (macro-action discovery), and low (primitive actions discovery). The high-level layer discovers a task representation y, using a recurrent unit, simultaneously with value-function training. The intermediate layer analyzes the representation given the current state to determine the macro-action z which is the (+/-) sign of actions. This is trained via VAE using the missing information technique. The state element \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego}$$\end{document} is a subset of the state set, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}^{ego} \in \mathcal {S}$$\end{document} that includes everything except the agent’s self-state. The subsequent loss in decoder with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego}$$\end{document} will shape macro-actions, z, task-agnostic and compact. The representation and macro-action are then conditioned in the low-level policy to make primitive decisions. Gradients do not overflow between hierarchical layers, ensuring each layer’s independent role. High-Level Layer The High-Level (HL) architecture aims to discover latent task representations as a probability distribution that reconciles the trajectory data across multiple tasks. Our HL module utilizes recurrent units for modeling the probability distributions. We use a categorical feed-forward network with an information-theoretic regularizer10 to induce explorations while penalizing allocations if it is not worthwhile. To align our experimental settings with the baselines in the Experiment section, we use a GRU11 as the recurrent unit, though an LSTM12 can also be an alternative. For simplicity, we denote the HL layer’s parameters by ϕ and use subscripts to indicate specific sub-architectures. For example, the recurrent model in Fig. 2 is represented as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi _h$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi _h$$\end{document} denotes the encoder giving output h. The HL feed-forward process is denoted given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {MDP}_t = (s_t, a_t, r_t, s_{t+1})$$\end{document}:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} h_t = \phi _h (\text {MDP}_t, h_{t-1}), \quad y_t = \text {Cat}(\phi _y(h_t)) \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h_{t-1} = \phi _h(\text {MDP}_{0:t-1})$$\end{document} is the hidden state of the recurrent unit, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi _y$$\end{document} is a feed forward neural network, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Cat}$$\end{document} is a categorical activation into discrete probability space. Algorithm 1HiMeta: Hierarchical Meta-RL HiMeta: Hierarchical Meta-RL HL-training scheme Our HL design includes an encoder where the representation is inferred while being simultaneously trained alongside a value function using the Mean Squared Error (MSE) loss:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}_{HL(V^{\pi })} = [V^{\pi _\theta }(\phi (MDP_t), s_t) - V(s_t)]^2 \end{aligned}$$\end{document}where the first term is our estimation with the learned representations, and the second is a true state value function. The true value function is estimated via Monte-Carlo samplings. Inspired by10, the entropy regularizer is used to encourage exploration of representations within a task \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {T}$$\end{document}, while the occupancy loss ensures having likely correct task representations only when it successfully reduces the value loss. Formally, the entropy regularizer is defined as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}_{HL(ent)} = \log \frac{p(y_t)}{\phi _y(y_t|h_t)} \end{aligned}$$\end{document}and the occupancy loss is:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \mathcal {L}_{HL(occ)} = - (\textbf{e}^T y_t)\log K \end{aligned} \end{aligned}$$\end{document}where K is the task inference dimension and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{e} = [ e^{-K+1}, e^{-K+2}, \dots , e^{-1}, e^0 ]\in \mathbb {R}^K$$\end{document}. We scale it with the upper bound of the entropy regularizer łog K, encouraging their competition. Thus, by creating the HL objective with scaling parameters, the high-level architecture can successfully and efficiently infer representations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_t$$\end{document} for use by the lower-level architectures. The HL update is to optimize the following loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}_{HL}$$\end{document} as9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \min _{\phi } \alpha _V \mathcal {L}_{HL(V^{\pi })} + \alpha _{\epsilon }\mathcal {L}_{HL(ent)} + \alpha _o\mathcal {L}_{HL(occ)} \end{aligned} \end{aligned}$$\end{document}The above training of the HL parameters ϕ is run across N tasks. Intermediate-level layer Our key contribution is the Intermediate-Level (IL) architecture. At this level, we infer a macro-action using a modified VAE. The objective is to further distribute the decision burden from the policy and guide the policy to the desired state space using goal-conditioning. This is fundamentally different from previous hierarchical methods that only aimed to reach the goal state while neglecting the complexity of learning systems such as multi-task and meta-learning. Our design leverages a missing-information technique, formally imputations13–15, which uses relevant data to infer the expected data pattern in a missing region. Given two arbitrary points in the state space where the actions to transition from one to the other are unknown, we aim to discover macro actions, z, that connect them. The term “macro-” is used as a temporally extended action: Definition 1 (Macro-action) Given a function f(· ), a macro-action, z, is defined as information that connects the current state to the desired goal state within the state space: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{t+M} = f(s_{t}, z)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M \in \mathbb {N}_+$$\end{document}. The overall feed-forward process for macro-action discovery by the encoder is as follows:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} z_t \sim \text {tanh}(\mathcal {N} (\psi _{z_\mu }(y_t, s_t), \psi _{z_\sigma }(y_t, s_t))), \end{aligned}$$\end{document}where a Multivariate Gaussian distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {N}(\cdot , \cdot )$$\end{document} is constructed with mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{z_\mu }(\cdot , \cdot )$$\end{document} and variance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{z_\sigma }(\cdot , \cdot )$$\end{document}. IL-training scheme Imputation here is to find a proper action space embedding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {Z} \approx \mathcal {A}$$\end{document} that satisfies \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S} \times \mathcal {Z}\rightarrow \mathcal {S}$$\end{document}. This is a complex problem, as the encoder and decoder have to concurrently bridge state and action spaces. Without proper treatment, Eq. (10) will collapse, providing meaningless information only to lower the loss. Therefore, we modify the existing architecture and loss function of VAE to not only perform the standard VAE operations but also bridge the two spaces by replacing the prior distribution p(z) in the Evidence Lower Bound (ELBO) with action distributions. We also provide the state as input to the decoder to make it a transition predictor. The architecture of the intermediate-level layer is depicted in Fig. 2. This strategy reshapes the latent space of the VAE by aligning the current belief, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _z(y_t, s_t)$$\end{document}, with the action space. In other words, the encoder, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S} \times \mathcal {Y}\rightarrow \mathcal {A}$$\end{document}, remains within the action space given the state in Eq. (10), while the decoder learns a function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f: \mathcal {S} \times \mathcal {A} \rightarrow \mathcal {S}$$\end{document} in Definition 1 that maps the state and action to the future state. This models the decoder as a cumulative transition function since it predicts M steps forward future state with current state and macro-actions. This shapes the macro-actions as compressed action embeddings within the action space and confined in the domain of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[-1, 1] \sim \tanh (\cdot )$$\end{document}. This represents our novel contribution, which learns the latent macro-action to directly assist the policy in reducing the decision-making challenge in complex multi-task execution using successful goal states from the policy. This prevents the loss of optimal behavior in some tasks while learning others. Additionally, we introduce the ego-state for task-agnostic inference by masking task-related state-wise elements: Definition 2 (Ego-state) Given a state set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}$$\end{document}, the set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}$$\end{document} consists of two subsets: ego-state and other-state, such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}_{ego} \oplus \mathcal {S}_{other} = \mathcal {S}$$\end{document}. The ego-state, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}_{ego}$$\end{document}, includes only self-state elements. Remark 1 An example of an ego state is a blind robot. For the robot that is to perform any arbitrary task (e.g., navigating, grasping, or reaching), the task-agnostic components are only its body-related components: the angle of joints and the relative arm/body position. While state space decomposition is beneficial for task-agnostic learning8, this instead voids out the necessary task information so that inference can stay in a high-level space instead of detailed execution. For our experiments on MetaWorld (manipulator-based environments), we define the ego state as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego} := (p_x, p_y, p_z, o)$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_x, p_y, p_z$$\end{document} denote the 3D Cartesian coordinates of the gripper and o indicates the gripper’s opening. The remainder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{other}$$\end{document} consists of the 3D Cartesian coordinates and quaternion of the task-specific objects. The ego-state is particularly appealing for distributing decision-making into a set of independent roles. Specifically, macro-actions provide high-level action embedding, while the policy executes the detailed actions. Finally, the macro-action is then trained over the encoder parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _z$$\end{document} and decoder parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _s$$\end{document} by minimizing the following loss:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \min _{\psi _z, \psi _s} \text {ELBO}_t(y_t, z_t, s_t, s^{ego}_t, s^{ego}_{t+M}) \\ \text {ELBO}_t(\cdot ) = \ \mathcal {L}_{IL(KL)} + \mathcal {L}_{IL(trans)} \\ \end{aligned} \end{aligned}$$\end{document} Given the decoder’s forward pass \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{t+M}^{ego} = \psi _s(y_t, z_t, s_t^{ego})$$\end{document}, each loss term is given below:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \mathcal {L}_{IL(KL)} =&\ \log \frac{p(a_t|y_t, z_t, s_t)}{\psi _z(z_t|y_t, s_t)} \\ \mathcal {L}_{IL(trans)} =&\ \sum _{t=0}^T \log p (\psi _s(s^{ego}_{t+M}|y_t, z_t, s^{ego}_t)) \\ \end{aligned} \end{aligned}$$\end{document} This approach is notably distinct from previous methods: SD10, VariBad16, LDM17, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {RL}^2$$\end{document}18, and PEARL19, as it involves shaping the prior distribution with the action space and incorporating additional ego-state input into the decoder to shape it as a transitional function. In Eq. (12), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{t+M}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s^{ego}$$\end{document} are goal state and ego-state respectively. The goal state is the state the policy aims to achieve, typically at the designer’s discretion, or it can be discovered using additional techniques. In our approach, we construct three definitions of goal state at the end of this section. Goal state generation The goal state, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{t+M}$$\end{document}, adapted per our method, is defined as follows: Constant discretization (CD): This naively discretizes the given episodic timelines into several segments. That is, we fix M as constant, thus obtaining a list of goal states: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{2M}, \dots , s_{nM}]$$\end{document}.Constant margins (CM): This method maintains a constant margin between the current state and the goal state, resulting in a list of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{1 + M} \dots s_{t + M}]$$\end{document}, given a constant M.Adaptive at sub-task (ST): This uses the inferred sub-task from the HL layer, analyzing when the label changes in timescale. For example, given an array of states \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S = [s_0, s_1, \dots , s_t]$$\end{document} and representations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y = [y_0, y_1, \dots , y_t]$$\end{document}, goal states are states where the representations change in time scale (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\exists t: [y_0:y_{t-1}] \ne [y_1:y_t] \rightarrow S[t]$$\end{document}).As noted, goal states heavily rely on the policy’s exploration. However, this lessens the information load by preventing the policy from forgetting previously learned tasks while it acquires new ones. In essence, although goal states are learned concurrently, this strategy ultimately helps the policy shift focus away from past tasks by relying on learned macro-actions for optimally learned tasks. In this work, we manually defined the ego state and the three goal states, based on the nature and prior knowledge of the agents/ tasks. However, exploring methods to automatically discover better state representations or abstractions is a promising direction for future research. Constant discretization (CD): This naively discretizes the given episodic timelines into several segments. That is, we fix M as constant, thus obtaining a list of goal states: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{2M}, \dots , s_{nM}]$$\end{document}. Constant discretization (CD): This naively discretizes the given episodic timelines into several segments. That is, we fix M as constant, thus obtaining a list of goal states: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{2M}, \dots , s_{nM}]$$\end{document}. Constant margins (CM): This method maintains a constant margin between the current state and the goal state, resulting in a list of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{1 + M} \dots s_{t + M}]$$\end{document}, given a constant M. Constant margins (CM): This method maintains a constant margin between the current state and the goal state, resulting in a list of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[s_{M}, s_{1 + M} \dots s_{t + M}]$$\end{document}, given a constant M. Adaptive at sub-task (ST): This uses the inferred sub-task from the HL layer, analyzing when the label changes in timescale. For example, given an array of states \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S = [s_0, s_1, \dots , s_t]$$\end{document} and representations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y = [y_0, y_1, \dots , y_t]$$\end{document}, goal states are states where the representations change in time scale (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\exists t: [y_0:y_{t-1}] \ne [y_1:y_t] \rightarrow S[t]$$\end{document}). Adaptive at sub-task (ST): This uses the inferred sub-task from the HL layer, analyzing when the label changes in timescale. For example, given an array of states \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S = [s_0, s_1, \dots , s_t]$$\end{document} and representations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y = [y_0, y_1, \dots , y_t]$$\end{document}, goal states are states where the representations change in time scale (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\exists t: [y_0:y_{t-1}] \ne [y_1:y_t] \rightarrow S[t]$$\end{document}). Low-Level Layer For policy training, which is the objective of the Low-Level (LL) layer, we chose Proximal Policy Optimization (PPO), a practical solution that effectively reduces sample complexity using an off-policy gradient clipping approach. We briefly introduce its idea here, leaving parametric details in the Appendix. PPO maximizes the following objectives:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \pi _\theta = \mathop {{{\,\mathrm{arg\,max}\,}}}\limits _\theta \mathbb {E} \bigl [ \mathcal {J}(\theta , \mathcal {T}) - \alpha _1 \mathcal {L}(\theta , \mathcal {S}) + \alpha _2 \mathcal {E}(\theta ) \bigr ] \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {J}(\theta , \mathcal {T})$$\end{document} is the gradient-clipped advantage for optimality, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}(\theta , \mathcal {S})$$\end{document} is the value loss as in Eq. (6), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {E}(\theta )$$\end{document} is the entropy loss for sufficient exploration, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _2$$\end{document} are scaling parameters. Note that our LL is trained without the value loss in Eq. 13 since it is trained with the HL objective (Eq. 6). The feed-forward inference of the LL layer is:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} a_t = \pi _\theta (y_t, z_t, s_t) \end{aligned} \end{aligned}$$\end{document} For effective and efficient collaboration between the macro-action and primitive-action, we additionally construct an intrinsic reward function that adds a bonus to the policy when it follows the given macro-action sign-wise (+/-). This assumes the action space spans both negative and positive spaces. MetaWorld, our testbed, has the range \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {A} \in [-1, 1]$$\end{document}. The reason for the sign-wise bonus is to assign independent roles in decision-making to macro-actions and primitive actions as high-level and low-level actions, respectively. The optimality objective, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {J}(\theta , \mathcal {T})$$\end{document}, includes additional intrinsic reward incurred achieving macro-actions, starting with the non-clipped advantage function below.15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A(s,a) = \textbf{r} + \gamma V(s') - V(s) \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{r} = r_{in} + r_{ext}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(s) = \textbf{r} + \gamma V(s')$$\end{document} To carefully design an intrinsic reward signal that does not disrupt the original task optimality, it is scaled with an extrinsic reward signal. The one-time step intrinsic reward function is defined as:16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{in} = r_{ext} \cdot \frac{1}{|\mathcal {A}|} \sum _\mathcal {A} \mathbb {I}\big [ sgn(z) = sgn(a) \big ] \end{aligned}$$\end{document} The intrinsic reward signal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{in}$$\end{document} is high only when the extrinsic reward signal is high and is bounded by the maximum value of the extrinsic reward. This enables the policy to maintain optimal behavior in tasks it has already mastered while still allowing for exploration in tasks it is still learning. Analysis Here, we provide a mathematical view of macro-action discovery. To understand the trajectory of the system, consider the state transition over a discrete time interval [t, t + M]. Denoting the transition function at time t as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_t(s_t, a_t) = s_{t+1}$$\end{document}, the trajectory from time t to t + M can be represented as a composite function of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_t(\cdot )$$\end{document}:17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} s_{t}, s_{t+1}, \cdots s_{t+M} = (T_{t+M-1} \circ \cdots \circ T_{t+1} \circ T_t)(s_t, a_t, \ldots , a_{t+M-1}) \end{aligned} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{t+M - 1} \circ \cdots \circ T_{t+1} \circ T_t$$\end{document} denotes the composition of transition functions with the input of state-action pairs between time t and t + M - 1. In a model-free setting, the objective is to determine the next action \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_{t+1}$$\end{document} based on the current state \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_t$$\end{document}, not the whole trajectory. Thus, we define a function approximator for learning such composite functions as in Eq. (17) with one single input:18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} s_{t+M} = f(s_t, z_t|\psi ) \end{aligned}$$\end{document}where the neural approximator ψ has at least two components for learning the composite function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(T_{t+M - 1} \circ \cdots \circ T_{t+1} \circ T_t)$$\end{document} with the decoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{s}$$\end{document}, and the action summary, z, of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(a_t, \cdots , a_{t + M - 1})$$\end{document} with the encoder \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _z$$\end{document}. This reflects the design intent of an intermediate layer with an encoder and decoder for learning the composite function and the macro-action, respectively. To further investigate the mathematical definition of z, we define a simple deterministic linear system:19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} s_{t+1} = s_t + B\pi (s_t) = T_t(s_t, \pi (s_t)) \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_t \subset \mathbb {R}^s$$\end{document} is the state at time t, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B \subset \mathbb {R}^{s\times a}$$\end{document} is for a linear transformation that maps action to state space, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi (s_t)\subset \mathbb {R}^a$$\end{document} is the action from policy, and T is the transition function. Then, the recursive definition of the goal state under this system is20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} s_{t + M} = s_t + B\left( \sum _{k=t}^{t+M-1} \pi (s_k)\right) \end{aligned}$$\end{document} In the above formulation, it is represented by the sum of the initial state and the transformation of the sum of actions, i.e., the macro-actions. Our insight in Definition 1 is to approximate cumulative action embeddings using a missing-information technique with neural approximators for nonlinear system dynamics. Challenges and Solutions We additionally addressed two major challenges to enhance overall performance. First, we tackle the curse of hierarchy, which refers to the mismatch in focus across layers. As the policy gathers new trajectories to maximize rewards, higher-level layers must adapt to new incoming dynamical information. This issue, caused by transitional shifts in hierarchical RL20, is mitigated through (1) a data buffer that refreshes with recent trajectories, (2) independent roles for each layer by designing each layer to achieve a specific goal and preventing gradient overflow off layers, and (3) streamlined architectures to avoid model complexity with which the higher-level models easily track the traces of low-level policy. Second, when learning multiple tasks, policies often prioritize easier tasks by neglecting difficult tasks, leading to higher sample complexity. To counter this, we adopt reward shaping with a monotonic logarithm function that boosts low rewards while maintaining the same increment rate beyond a specified point. This makes early rewards for difficult tasks emphasized. For any non-negative rewards, the shaping function is:21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} g(x) = {\left\{ \begin{array}{ll} \ln \left( \frac{(e^a - a)}{a}x + 1\right) , & \text {if } x \le a, \\ \frac{a}{a^2 + 1}(x - a) + a, & \text {otherwise}. \end{array}\right. } \end{aligned}$$\end{document}where a is a hyperparameter within the reward range. The reshaped rewards, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g(R(s_t, a_t))$$\end{document}, are visualized with a=3 in Fig. 3.Fig. 3Reward shaping function with a=3. Rewards below 3 are exponential, while the gradient is linear beyond a=3. Reward shaping function with a=3. Rewards below 3 are exponential, while the gradient is linear beyond a=3. Code Availability The code that was used in the current study is available at https://github.com/Mgineer117/himeta. Experiments We utilized MetaWorld21 as our testbed. MetaWorld provides a structured environment for multi-task and meta-learning, designed to evaluate algorithms on their generalization capabilities. It improves upon earlier frameworks by systematically defining tasks with shared characteristics. We compared our approach against the following baselines:Learning tasks via subtask decomposition (SD)10Probabilistic embeddings for actor-critic RL (PEARL)19Note that SDVT10 combines SD with virtual training (VT), which adds computational complexity with minimal performance gains. Thus, we omit VT here for simplicity and it can be implemented per our method as well.Fig. 4MetaWorld ML10 learning curve is shown here. The mean and standard deviations are plotted along the average success and return metrics. (a) Cumulative success is plotted where success is set to 1 at the point the agent completes the task and afterward. (b) Cumulative rewards with γ = 1. Learning tasks via subtask decomposition (SD)10 Learning tasks via subtask decomposition (SD)10 Probabilistic embeddings for actor-critic RL (PEARL)19 Probabilistic embeddings for actor-critic RL (PEARL)19 MetaWorld ML10 learning curve is shown here. The mean and standard deviations are plotted along the average success and return metrics. (a) Cumulative success is plotted where success is set to 1 at the point the agent completes the task and afterward. (b) Cumulative rewards with γ = 1. SD is selected for its recent superior performance in MetaWorld, and our method builds on PEARL by incorporating an intermediate-level layer and replacing the encoder with a recurrent unit. Detailed network parameters and hyperparameters are provided in the Appendix.Table 1All experiments were conducted under the same parameters settings for a detailed comparison. The table shows average success, previously used success metrics, and returns with means and standard deviations. The previous success rate, ranging from 0 to 1, is considered 1 for the entire trajectory regardless of when the agent completed the task. In contrast, ours measures average success through each time step of the trajectory, highlighting our method is more effective since the increment from average success to success is comparatively smaller than the increment in the SD. Additionally, returns represent cumulative rewards with γ =1. These results suggest that our approach achieves significant performance gains by distributing information load and utilizing action-guided learning with macro-actions. The specific performance for each task is attached in the Tables 5 and 2.ML10Avg. successSuccessReturnsMethodsTrainTestTrainTestTrainTestHiMeta-CD\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.64\pm 0.07}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.13\pm 0.11}$$\end{document}0.73± 0.09\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.30\pm 0.10}$$\end{document}3232± 207\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {501\pm 361}$$\end{document}HiMeta-CM0.64± 0.100.12± 0.10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73\pm 0.07}$$\end{document}0.29± 0.12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {3306\pm 309}$$\end{document}485± 344HiMeta-ST0.64± 0.080.12± 0.080.72± 0.130.29± 0.113237± 296497± 335SD100.42± 0.090.05± 0.060.60± 0.040.06± 0.132438± 468422± 290PEARL19||0.22± 0.260.01± 0.021034± 1201340± 494 Table 1All experiments were conducted under the same parameters settings for a detailed comparison. The table shows average success, previously used success metrics, and returns with means and standard deviations. The previous success rate, ranging from 0 to 1, is considered 1 for the entire trajectory regardless of when the agent completed the task. In contrast, ours measures average success through each time step of the trajectory, highlighting our method is more effective since the increment from average success to success is comparatively smaller than the increment in the SD. Additionally, returns represent cumulative rewards with γ =1. These results suggest that our approach achieves significant performance gains by distributing information load and utilizing action-guided learning with macro-actions. The specific performance for each task is attached in the Tables 5 and 2.ML10Avg. successSuccessReturnsMethodsTrainTestTrainTestTrainTestHiMeta-CD\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.64\pm 0.07}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.13\pm 0.11}$$\end{document}0.73± 0.09\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.30\pm 0.10}$$\end{document}3232± 207\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {501\pm 361}$$\end{document}HiMeta-CM0.64± 0.100.12± 0.10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73\pm 0.07}$$\end{document}0.29± 0.12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {3306\pm 309}$$\end{document}485± 344HiMeta-ST0.64± 0.080.12± 0.080.72± 0.130.29± 0.113237± 296497± 335SD100.42± 0.090.05± 0.060.60± 0.040.06± 0.132438± 468422± 290PEARL19||0.22± 0.260.01± 0.021034± 1201340± 494 All experiments were conducted under the same parameters settings for a detailed comparison. The table shows average success, previously used success metrics, and returns with means and standard deviations. The previous success rate, ranging from 0 to 1, is considered 1 for the entire trajectory regardless of when the agent completed the task. In contrast, ours measures average success through each time step of the trajectory, highlighting our method is more effective since the increment from average success to success is comparatively smaller than the increment in the SD. Additionally, returns represent cumulative rewards with γ =1. These results suggest that our approach achieves significant performance gains by distributing information load and utilizing action-guided learning with macro-actions. The specific performance for each task is attached in the Tables 5 and 2. Evaluation Metric MetaWorld introduced a success metric to complement reward-based evaluations. In our settings, we use an average episodic success metric that not only indicates task success but also measures how quickly the task is accomplished. Unlike21, which reports the maximum success rate achieved over the training process, or10, which measures success at the final time step without considering the time taken, our metric provides a more nuanced view of performance. In particular, we measure how macro-actions effectively guide policy via time-wise success considerations. However, we provide additional results for completeness using the previous success metric in10.Table 2Task-wise success rates with mean and standard deviations.Average success (train)TaskHiMeta-CDHiMeta-CMHiMeta-STSDPEARLReach\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73 \pm 0.11}$$\end{document}0.64 ± 0.30.68 ± 0.120.34 ± 0.130.46 ± 0.04Push\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73 \pm 0.04}$$\end{document}0.69 ± 0.150.65 ± 0.100.07 ± 0.110.0 ± 0.0Pick-place0.55 ± 0.26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.70 \pm 0.14}$$\end{document}0.55 ± 0.260.03 ± 0.050.0 ± 0.0Door-open0.77 ± 0.040.72 ± 0.06\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.78 \pm 0.02}$$\end{document}0.76 ± 0.050.25 ± 0.03Drawer-close\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}0.95 ± 0.00.74 ± 0.07Button-press0.76 ± 0.060.71 ± 0.080.80 ± 0.06\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.88 \pm 0.01}$$\end{document}0.35 ± 0.03Peg-insert-side0.78 ± 0.03\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.80 \pm 0.03}$$\end{document}0.79 ± 0.030.02 ± 0.020.0 ± 0.0Window-open0.87 ± 0.02\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.88 \pm 0.01}$$\end{document}0.87 ± 0.020.86 ± 0.010.43 ± 0.1Sweep0.71 ± 0.07\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.75 \pm 0.04}$$\end{document}0.72 ± 0.080.33 ± 0.180.0 ± 0.0Basketball0.03 ± 0.030.05 ± 0.060.07 ± 0.08\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.30 \pm 0.30}$$\end{document}0.0 ± 0.0Average success (test)Drawer-open0.15 ± 0.150.17 ± 0.17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.17 \pm 0.13}$$\end{document}0.10 ± 0.140.02 ± 0.01Door-close\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.21 \pm 0.19}$$\end{document}0.16 ± 0.100.19 ± 0.090.10 ± 0.080.03 ± 0.02Shelf-place\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.03 \pm 0.03}$$\end{document}0.0 ± 0.00.01 ± 0.020.0 ± 0.00.0 ± 0.0Sweep-into\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.29 \pm 0.20}$$\end{document}0.29 ± 0.270.27 ± 0.200.11 ± 0.140.0 ± 0.0Lever-pull0.0 ± 0.0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.02 \pm 0.03}$$\end{document}0.0 ± 0.00.01 ± 0.020.0 ± 0.01 Table 2Task-wise success rates with mean and standard deviations.Average success (train)TaskHiMeta-CDHiMeta-CMHiMeta-STSDPEARLReach\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73 \pm 0.11}$$\end{document}0.64 ± 0.30.68 ± 0.120.34 ± 0.130.46 ± 0.04Push\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.73 \pm 0.04}$$\end{document}0.69 ± 0.150.65 ± 0.100.07 ± 0.110.0 ± 0.0Pick-place0.55 ± 0.26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.70 \pm 0.14}$$\end{document}0.55 ± 0.260.03 ± 0.050.0 ± 0.0Door-open0.77 ± 0.040.72 ± 0.06\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.78 \pm 0.02}$$\end{document}0.76 ± 0.050.25 ± 0.03Drawer-close\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.96 \pm 0.0}$$\end{document}0.95 ± 0.00.74 ± 0.07Button-press0.76 ± 0.060.71 ± 0.080.80 ± 0.06\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.88 \pm 0.01}$$\end{document}0.35 ± 0.03Peg-insert-side0.78 ± 0.03\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.80 \pm 0.03}$$\end{document}0.79 ± 0.030.02 ± 0.020.0 ± 0.0Window-open0.87 ± 0.02\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.88 \pm 0.01}$$\end{document}0.87 ± 0.020.86 ± 0.010.43 ± 0.1Sweep0.71 ± 0.07\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.75 \pm 0.04}$$\end{document}0.72 ± 0.080.33 ± 0.180.0 ± 0.0Basketball0.03 ± 0.030.05 ± 0.060.07 ± 0.08\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.30 \pm 0.30}$$\end{document}0.0 ± 0.0Average success (test)Drawer-open0.15 ± 0.150.17 ± 0.17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.17 \pm 0.13}$$\end{document}0.10 ± 0.140.02 ± 0.01Door-close\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.21 \pm 0.19}$$\end{document}0.16 ± 0.100.19 ± 0.090.10 ± 0.080.03 ± 0.02Shelf-place\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.03 \pm 0.03}$$\end{document}0.0 ± 0.00.01 ± 0.020.0 ± 0.00.0 ± 0.0Sweep-into\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.29 \pm 0.20}$$\end{document}0.29 ± 0.270.27 ± 0.200.11 ± 0.140.0 ± 0.0Lever-pull0.0 ± 0.0\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {0.02 \pm 0.03}$$\end{document}0.0 ± 0.00.01 ± 0.020.0 ± 0.01 Task-wise success rates with mean and standard deviations. The evaluation is conducted using ML-10. This consists of 10 training tasks and 5 test tasks, expecting representations learned from training tasks to be transferred to performance in testing tasks. Sampling is based on 5 episodic samples per iteration for each task (a total of 50 trajectories). Given each episode lasting 500-time steps, 7.5k iterations are taken for training. Evaluation We performed all evaluations using five random seeds, maintaining fixed algorithmic and environmental parameters while accounting for variations in stochastic sampling and policy optimization. This also shows the statistical significance of the comparison. The quantitative results, including average training and testing success rates alongside cumulative rewards, are summarized in Table 1, with the corresponding learning curves illustrated in Fig. 4. For our ablation studies, we focus exclusively on the goal state representations detailed in “Intermediate-level layer”. We leave the investigation of further goal-related and other ablations as a promising direction for future work. One may notice a discrepancy between our experimental results and those in10. This is attributed to the MetaWorld version, which underwent a significant update in 2023. Additional remarks are provided in the Appendix. For completeness, task-specific learning results are included in Tables 2 and 5. While our methods with three different goal states do not exhibit significant benefits over other goals, they consistently outperform the considered baselines in terms of both success and return metrics under the same parametric and learning settings. Specifically, the choice of different goal states is robust, as they do not exhibit significant performance differences. A systematic investigation into stress-testing and identifying “failure mode” goal configurations that might degrade performance remains an objective for future study. Moreover, the superior performance of our algorithm—enabled by the intermediate layer and its associated modifications over PEARL—demonstrates the value of this intermediate layer. The high variance in PEARL suggests that it can learn some tasks while struggling with others. The latest state-of-the-art method, SD10, which has outperformed all previous meta-learning algorithms (VariBad16, LDM17, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {RL}^2$$\end{document}18, and PEARL19), falls short compared to our approach. Furthermore, the smaller gap of our method from average success to success in Table 1 indicates a faster task completion rate by the agent. In this context, our approach achieves state-of-the-art performance and efficiency in completing tasks. Related Works Parameter-sharing approaches explore “which model components to share” across tasks. Based on metadata like task labels, models determine the extent of weight and bias sharing, making non-shared parameters task-specific. Fully shared parameter architectures are termed “hard-parameter sharing”, while those with varying degrees of task-specific architectures are “soft-parameter sharing”. This approach proves effective in environments with slight deviations across tasks. Specifically, Ref.22 proposed a multi-layer encoder architecture composed of three distinct layers that can be either shared or unshared across tasks. The optimal degree of parameter sharing is then determined based on performance on individual tasks. In contrast, Ref.23,24 investigate adaptive approaches: the former adapts a base model to a new task by modifying a small, task-specific subset of parameters, while the latter determines which components of the existing model should be shared for the new tasks. These approaches differ fundamentally from ours: while they focus on deciding which components of a model to share across tasks, our method instead infers task-specific labels that identify the underlying task and then learns how to solve that task accordingly. Task grouping, in contrast, focuses on identifying tasks that can effectively share parameters by forming task groups. Unlike parameter-sharing, which assigns distinct parameters to each task, task-grouping simplifies model structures by grouping similar tasks together and treating them as a single task. Specifically, Ref.25 uses multiple encoders, each capturing different aspects of task similarity to best group similar tasks so that a single neural network can achieve strong performance on each group. In a similar vein, Ref.26 examines task grouping by analyzing gradient similarity, forming groups based on how closely task gradients align. Reference27 takes a different direction by using a neural network to predict the performance gain of adding a task to a group. Despite their differences, these methods all require training separate policies for each task group. In contrast, our algorithm is capable of handling diverse tasks that may belong to different groups without the need to train multiple policies. Consequently, relevant to ours, several works have attempted to discover shared structures across tasks, z, and condition them into the input space, π (a|s,z). Instead of merely expanding the input space, this method enriches it with relevant task-wise information, merging operational spaces that share similarities and significantly reducing the information load on the policy. Reference19 was a pioneering work that discovered latent task-wise information via posterior sampling by training alongside the value function. Additionally, Refs.10,16 utilize recurrent encoder and dynamic decoder architectures for representation learning through state and reward dynamical predictions. This approach effectively discovers task-wise latent information, integrates similar tasks, and distinguishes different ones, facilitating multi-task and meta-learning. In this context, our algorithm also addresses the decision burden—namely, the challenge that information from diverse tasks cannot be effectively distilled into a single policy. To mitigate this, we expand the input space using a tri-level framework consisting of task representation, macro-action discovery, and primitive-action discovery. Discussions Our approach adopts a tri-level framework that decomposes problem complexity into high-level task representation, intermediate-level macro-action discovery, and low-level primitive action discovery. While this structure yields strong performance on challenging meta-learning tasks, it introduces an additional complexity that can increase training times, cause instability, and induce more implementation difficulty. However, our proposed multi-layered architecture significantly reduces the problem complexity, enabling the policy to solve the task more efficiently, as the comparative experimental results show against the baselines. We acknowledge that our formal analysis is currently restricted to deterministic linear systems, which may imply that the strong performance of our algorithm may be tied to the specific experimental settings we evaluated. Additionally, our empirical evaluations are performed using robotic manipulation tasks, i.e., in MetaWorld, which is a standard testbed for meta?RL. We leave evaluating the scalability of our approach, or relevant variant, across more complex environments or robotics hardware as a promising avenue for future research. In reinforcement learning, tackling complex domains, such as those with high-dimensional image observations, conflicting objectives, or intricate reward functions, often requires discovering better state or goal abstractions to effectively guide policy optimization. Our algorithm operates on a similar principle by providing task-specific labels and high-level macro-action embeddings to address complex challenges like meta-RL. Because our approach to macro-action discovery offers a novel alternative to traditional state or goal abstractions, extending this action-centric embedding to solve a broader range of complex tasks represents a highly promising direction for future work. Additionally, future investigations may include offline multi-task or meta-learning28–30, safe reinforcement learning settings31,32, and multi-modal learning scenarios33,34. In such settings, we anticipate that proposed macro-actions will alleviate the problem-complexity due to difficult problem assumptions, additional constraints, and additional prediction features in one place. Conclusions Our approach introduces a novel hierarchical decomposition to alleviate the decision-making burden of policies in the context of hard-parameter sharing. We propose macro-actions, which provide compressed action information linking the current state to the handcrafted goal state. This is achieved by learning to produce both macro-actions and a composite transition function using a modified VAE. Our method demonstrates quicker task completion and outperforms previous baselines, which were known for their strong performance in training (multi-task) and test tasks (meta-learning). This advantage is attributed to our innovative approach to directly help the policy with an additional action embedding module that prevents the loss of optimality in learned tasks while simultaneously learning others.