HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym

Ngoc La, Ruaridh Mon-Williams, and Julie A. Shah · 2025

Read HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym by Ngoc La, Ruaridh Mon-Williams, and Julie A. Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22597.txt.

Introduction

Ngoc La and co-authors present HDDLGym, a Python tool that turns hierarchical planning problems into training grounds for reinforcement learning agents. Planning languages can state tasks as layers of subgoals, a structure that suits model based learning. Yet no tool bridged that world and the standard Gym test bed, as their 2025 paper notes.

HDDLGym reads domain and problem files in the HDDL planning language and builds Gym environments on its own. It supports many agents planning together, not just one. The paper walks through use with contest domains such as Transport, and shows how to craft new multi agent domains. A cooking game domain, Overcooked, demos the practical side.

Catalog record

CREATORNgoc La, Ruaridh Mon-Williams, and Julie A. Shah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22597

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym Ngoc La, Ruaridh Mon-Williams, and Julie A. Shah Abstract In recent years, reinforcement learning (RL) methods have been widely tested using tools like OpenAI Gym, though many tasks in these environments could also benefit from hierarchical planning. However, there is a lack of a tool that enables seamless integration of hierarchical planning with RL. Hierarchical Domain Definition Language (HDDL), used in classical planning, introduces a structured approach well-suited for model-based RL to address this gap. To bridge this integration, we introduce HDDLGym, a Python-based tool that automatically generates OpenAI Gym environments from HDDL domains and problems. HDDLGym serves as a link between RL and hierarchical planning, supporting multi-agent scenarios and enabling collaborative planning among agents. This paper provides an overview of HDDLGym’s design and implementation, highlighting the challenges and design choices involved in integrating HDDL with the Gym interface, and applying RL policies to support hierarchical planning. We also provide detailed instructions and demonstrations for using the HDDLGym framework, including how to work with existing HDDL domains and problems from International Planning Competitions, exemplified by the Transport domain. Additionally, we offer guidance on creating new HDDL domains for multi-agent scenarios and demonstrate the practical use of HDDLGym in the Overcooked domain. By leveraging the advantages of HDDL and Gym, HDDLGym aims to be a valuable tool for studying RL in hierarchical planning, particularly in multi-agent contexts. 1 MIT 2 University of Edinburgh [email protected], [email protected], [email protected] Code — https://github.com/HDDLGym/HDDLGym 1 Introduction Hierarchical planning is essential for addressing complex, long-horizon planning problems by decomposing them into smaller, manageable subproblems. In reinforcement learning (RL), hierarchical strategies can guide exploration along specific pathways, potentially enhancing learning efficiency. However, implementing RL policies within hierarchical frameworks often requires custom modifications to the original environments to incorporate high-level actions ( Wu et al. 2021 ; Liu et al. 2017 ; Xiao, Hoffman, and Amato 2020 ) . For example, in a Bayesian inference study using the Overcooked game, subtasks are integrated as high-level actions through specific rules embedded in the system codebase ( Wu et al. 2021 ) . Similarly, several RL studies use author-defined high-level actions, or macro-actions, to organize complex tasks ( Liu et al. 2017 ; Xiao, Hoffman, and Amato 2020 ) . While these studies highlight the benefits of hierarchical approaches in complex scenarios, the additional programming required to integrate hierarchical layers can make it challenging for external users to modify or implement alternative high-level strategies. This limitation reduces users’ flexibility to implement diverse hierarchical strategies tailored to their specific requirements. The Hierarchical Domain Definition Language (HDDL) ( Höller et al. 2020 ) is an extension of Planning Domain Definition Language (PDDL) ( McDermott et al. 1998 ) that incorporates hierarchical task networks (HTN) ( Erol, Hendler, and Nau 1994 ) . HDDL provides a standardized language for hierarchical planning systems and is supported by extensive documentation as well as a variety of domains and problems. Many of these resources are sourced from the hierarchical task network tracks of the International Planning Competitions (IPC-HTN) ( IPC 2023 HTN Tracks ) . HDDL’s intuitive and flexible design also allows users to define or modify problem-solving approaches by adjusting the hierarchical task networks to suit their specific needs. To leverage HDDL’s strengths in studying RL within hierarchical planning problems, we present HDDLGym — a framework that integrates HDDL with OpenAI Gym ( Brockman et al. 2016 ) , a standardized RL interface. HDDLGym is a Python-based tool that automatically generates Gym environments from HDDL domain and problem files. Multi-agent contexts are a key area in automated planning and hierarchical planning research. While HDDL is not inherently designed for multi-agent systems, multi-agent features have been explored in planning formalisms like MA-PDDL ( Kovacs 2012 ) and MA-HTN ( Cardoso and Bordini 2017 ) . However, to utilize the extensive, well-documented HDDL domains and problems from IPC-HTN, HDDLGym is designed to work closely with the HDDL defined by Höller et al. (2020) . We introduce a new protocol for extending HDDL domains and problems to support HDDLGym with multi-agent features. This includes making minor modifications to existing HDDL files from IPC-HTN. Main contributions This paper makes the following three key contributions: • We introduce HDDLGym, a novel framework that automatically bridges reinforcement learning and hierarchical planning by automatically generating Gym environments from HDDL domains and problems. • We provide a protocol for modifying HDDL domains to support multi-agent configurations within HDDLGym, thereby extending hierarchical planning techniques to complex multi-agent environments. • We detail HDDLGym’s design and usage, demonstrating its effectiveness with examples from the Transport domain (in IPC-HTN) and the Overcooked environment (as shown in Figures 1(a) and 1(b) , respectively). Core features HDDLGym offers five core features, which are detailed throughout the paper. • Support verifying and adapting HDDL files to the tool (Section 4 ); • Support multi-agent collaborations (Sections 5.5 and 6 ); • Support centralized and decentralized planning (Section 5.5 ); • Support modifying design of RL policy in hierarchical planning (Sections 5.3 and 5.4 ); • Support training and deploying trained RL policies with various evaluation metrics and visualization (Sections 6 ). The remainder of this paper is organized as follows. Section 2 provides background information on HDDL and OpenAI Gym, the two foundational frameworks on which our system is built. Section 3 discusses relevant prior work and thus highlights our contributions to the field. Section 4 then introduces the formal framework of HDDLGym, detailing how HDDL is modified to align with the agent-centric design of this tool. Section 5 covers the design and implementation details of HDDLGym. Following this, section 6 demonstrates the use of HDDLGym with examples from the Transport domain, representing domains from IPC-HTN, and Overcooked, representing customized environments. Section 7 discusses the key benefits and current limitations of the HDDLGym tool, along with future developments to address these limitations and expand its applications within artificial intelligence research. Finally, Section 8 concludes the paper. (a) Transport scenario (b) Overcooked scenario Figure 1: Examples of the Transport and Overcooked environments in HDDLGym 2 Background 2.1 HDDL HDDL ( Höller et al. 2020 ) is an extension of PDDL ( McDermott et al. 1998 ) . Höller et al. (2020) define the domain and problem as follows. Definition of Planning Domain: A planning domain D is a tuple (L,T_{P},T_{C},M) defined as follows. • L is the underlying predicate logic. • T_{P} and T_{C} are finite sets of primitive and compound tasks, respectively. • M is a finite set of decomposition methods with compound tasks from T_{C} and task networks over the set T_{P}\cup T_{C} . Definition of Planning Problem: A planning problem \mathcal{P} is a tuple (D,s_{I},tn_{I},g) , where: • s_{I}\in S is the initial state, a ground conjunction of positive literals over the predicates assuming the closed world assumption. • tn_{I} is the initial task network that may not necessarily be grounded. • g is the goal description, being a first-order formula over the predicates (not necessarily ground). In other words, beyond the action definition in PDDL, which establishes the rules of interaction with the environment, HDDL introduces two additional operators: task and method . In

Read the complete text