SHARPIE: A Modular Framework for Reinforcement Learning and Human-AI Interaction Experiments
Hüseyin Aydın, Kevin Godin-Dubois, Libio Goncalvez Braz, Floris den Hengst, Kim Baraka, Mustafa Mert Çelikok, Andreas Sauter, Shihan Wang, Frans A. Oliehoek · 2025
Read SHARPIE: A Modular Framework for Reinforcement Learning and Human-AI Interaction Experiments by Hüseyin Aydın, Kevin Godin-Dubois, Libio Goncalvez Braz, Floris den Hengst, Kim Baraka, Mustafa Mert Çelikok, Andreas Sauter, Shihan Wang, Frans A. Oliehoek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19245.txt.
Introduction
Huseyin Aydin and co-authors wrote a 2025 paper on SHARPIE, a shared platform for experiments where people work with learning agents. Reinforcement learning fits such teamwork, but each lab built its own rig. SHARPIE offers common parts instead: wrappers for environments, a web face for participants, logs, and cloud links.
Its design also ties into crowdsourcing sites for recruiting. The range of studies it can host is wide. Topics span reward shaping with people, learning from feedback, handing off actions, and teaming. A shared interface, the authors argue, can standardise a young field. Comparable setups make results easier to trust and repeat.
Catalog record
| CREATOR | Hüseyin Aydın, Kevin Godin-Dubois, Libio Goncalvez Braz, Floris den Hengst, Kim Baraka, Mustafa Mert Çelikok, Andreas Sauter, Shihan Wang, Frans A. Oliehoek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SHARPIE: A Modular Framework for Reinforcement Learning and Human-AI Interaction Experiments Hüseyin Aydın, Kevin Godin-Dubois, Libio Goncalvez Braz, Floris den Hengst, Kim Baraka, Mustafa Mert Çelikok, Andreas Sauter, Shihan Wang, Frans A. Oliehoek Abstract Reinforcement learning (RL) offers a general approach for modeling and training AI agents, including human-AI interaction scenarios. In this paper, we propose SHARPIE ( S hared H uman- A I R einforcement Learning P latform for I nteractive E xperiments) to address the need for a generic framework to support experiments with RL agents and humans. Its modular design consists of a versatile wrapper for RL environments and algorithm libraries, a participant-facing web interface, logging utilities, deployment on popular cloud and participant recruitment platforms. It empowers researchers to study a wide variety of research questions related to the interaction between humans and RL agents, including those related to interactive reward specification and learning, learning from human feedback, action delegation, preference elicitation, user-modeling, and human-AI teaming. The platform is based on a generic interface for human-RL interactions that aims to standardize the field of study on RL in human contexts. Code — https://github.com/libgoncalv/SHARPIE Introduction Reinforcement learning (RL) refers to a family of algorithms in which agents learn from the consequences of their own interactions with an environment to try to maximize the long-term reward obtained ( Sutton and Barto 1998 ) . The available compute has seen an increase in the use of RL methods in a wide variety of problems, many of which involve humans in some sense ( Den Hengst et al. 2020 ) . RL agents interact with humans in a wide variety of ways: in some problem settings, RL agents observe humans to achieve a common goal or act on their behalf ( Natarajan et al. 2010 ) ; in other settings, RL agents communicate with humans to provide services or support their decision making ( Zhao et al. 2021 ) ; human feedback can also be used to evaluate the value of the agents’ actions in other settings ( Knox and Stone 2009 ; Christiano et al. 2017 ) . These different types of interaction highlight the need to incorporate humans in the training and evaluation of RL agents. However, their interaction patterns remain limited to basic and unidirectional. In contrast, hybrid human-AI intelligence, as envisioned by Akata et al. 2020 emphasizes the need for rich, diverse, and dynamic interaction patterns between humans and AI agents in order to effectively address problems which humans nor agents cannot solve independently. Although there are software packages to study the interaction of RL agents and their environments, little attention has been paid to training and evaluation of RL agents in diverse and dynamic interaction patterns that include both humans and other agents. In particular, popular RL libraries such as Gymnasium ( Brockman 2016 ; Towers et al. 2024 ) , PettingZoo ( Terry et al. 2021 ) , JaxMARL ( Rutherford et al. 2024 ) , MO-Gymnasium ( Alegre et al. 2022 ) or HIPPO-Gym ( Taylor et al. 2023 ) , provide a framework to train RL agents in benchmarks of particular formulations of the RL problem. These libraries also provide an interface to include a custom domain in the training. However, none of these libraries by themselves specifically tackles rich, diverse and dynamic interactions among humans and RL agents such as multi-modal communications for shared observations, delegated actions and common goals that we believe to be required for hybrid intelligence. To address this research gap, we propose SHARPIE ( S hared H uman- A I R einforcement-Learning P latform for I nteractive E xperiments) as a platform for studying how multiple humans and RL agents interact and collaborate effectively. This modular platform streamlines the design and execution of experiments with humans and interactive RL agents by: • providing a versatile wrapper for popular RL, multi-agent RL, and multi-objective RL environments and algorithms • supporting configurable communication channels between the human and the RL agents with various modalities • offering logging services, deployment utilities, and participant recruitment platforms integration. The platform aims to empower researchers to address research questions on the interaction of RL agents and humans, including those related to e.g. interpretability, interactive reward specification and learning, learning from human feedback, action delegation, preference elicitation, user-modeling, and human-AI teaming. As SHARPIE aims to facilitate human-agent collaboration research in the most generic way possible, it may also prove useful for simulating interactive robotic tasks prior to their physical implementation. Hence, we believe that SHARPIE will also lead to new research questions in the field of human-robot interaction ( Goodrich, Schultz et al. 2008 ) . Apart from its potential impact on other fields, the main contribution of SHARPIE is to introduce a standard for RL-based human-agent interactions in a multi-agent setting in the same way that the Gymnasium API has become this standard for fully simulated RL environments. In this paper, we first present some motivating use cases to sketch the scope and intended outcomes of this project with tangible use cases, including their relation with cognitive science. Next, we describe the SHARPIE framework design principles, the challenges encountered, and the solutions implemented to tackle these. We close with a discussion of future ambitions. Motivating Use Cases Human role(s) Agent role(s) Environment Interface 1 Reward annotation 1 annotator 1 learner AMaze (discrete) 1 Video output with timed feedback 2 Exploration 1 task specifier n learners AMaze (hybrid) 2 Maze layout & policy selection 3 Teaching n teachers n students Simple Spread 3 Demonstrations 4 Action delegation 1 controller 1 learner Mountain Car 4 Environment interactions 5 Task specification 1 task specifier 1 learner MineCraft 5 Text communication channel 6 Human-AI Teaming n collaborators Simple Tag 3 Environment interactions, intentions 7 Utility elicitation 1 stakeholder 1 learner Deep Sea Treasure 6 Example policies 8 Shared decision support n stakeholders 1 advisor Water-reservoir 7 Environment interactions Table 1: Use case examples. References are: 1- Godin-Dubois, Miras, and Kononova 2024 ; 2- Godin-Dubois, Miras, and Kononova 2025 ; 3- Lowe et al. 2017 ; 4- Moore 1990 ; 5- Johnson et al. 2016 ; 6- Vamplew et al. 2011 ; 7- Pianosi, Castelletti, and Restelli 2013 . Table 1 provides practical examples of target use cases that illustrate the wide range of problems that SHARPIE is designed to deal with. It details for each use case the roles taken on by human(s) and agent(s), an example environment and an envisioned interaction interface. These example use cases are not comprehensive and we will only implement some of these initially. We provide them here for illustrative purposes. The first use case, entitled ‘reward annotation’, corresponds to a setting in which a human annotator supplies the agent with a dense reward signal based on observations of the agent interacting with an environment. This reward signal complements a sparse reward signal in a maze navigation task, and models a setting in which a human expert guides the agent in completing its task successfully. It requires a visual representation of the environment and human reward feedback components in the User Interface (UI). The second, ‘exploration’, use case targets the same maze solving task but suggests richer platform capabilities, as it requires the