Adaptive Human–AI Coordination via Hierarchical Action Disentanglement

Adnan Ahmad [email protected], Bahareh Nakisa [email protected], Mohammad Naim Rastgoo [email protected] · 2026

Read Adaptive Human–AI Coordination via Hierarchical Action Disentanglement by Adnan Ahmad [email protected], Bahareh Nakisa [email protected], Mohammad Naim Rastgoo [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24343.txt.

Introduction

Ahmad, Nakisa, and Rastgoo propose Intrinsic Action Disentanglement, a layered learning frame for human-AI teamwork. Team agents must adapt to diverse partner conduct and skill levels, but older methods collapse to one style. Intrinsic Action Disentanglement learns distinct, partner-aware action sequences tied to latent skills. An inner reward pushes split action patterns across skills, giving a clear map from choices to replies. Catching long play runs, it adapts freely to mixed partner shifts. In Overcooked-AI across layouts and partner settings, it beats strong rivals.

Catalog record

CREATORAdnan Ahmad [email protected], Bahareh Nakisa [email protected], Mohammad Naim Rastgoo [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47052
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Human–AI Coordination via Hierarchical Action Disentanglement Adnan Ahmad [email protected], Bahareh Nakisa [email protected], Mohammad Naim Rastgoo [email protected] Abstract Human–AI collaboration requires agents that can adapt to diverse partner behaviors and skill levels while remaining robust to unseen partners. Existing methods often collapse to a single dominant behavior or learn poorly aligned skills, limiting effective coordination. We propose Intrinsic Action Disentanglement (IAD), a deep hierarchical reinforcement learning (DHRL) framework that learns distinct, partner-aware low-level action sequences conditioned on high-level latent skills. IAD introduces an intrinsic reward that explicitly encourages disentangled action distributions of the agent’s low-level policy across skills, yielding an interpretable mapping between high-level decisions and partner-specific behavioral responses. By capturing temporally extended interaction patterns, IAD enables flexible adaptation to heterogeneous partner dynamics under distributional shift. We evaluate IAD in the Overcooked-AI domain across multiple layouts and diverse partner settings, including unseen simulated partners, a human-proxy model trained on human–human gameplay, and real human partners. Results show that IAD consistently outperforms strong baselines and achieves more reliable, adaptive coordination across all settings. 1 Introduction Developing intelligent agents that can collaborate effectively with human partners remains a major challenge in multi-agent reinforcement learning (MARL) 14 ; 1 ; 3 . While MARL progress is often demonstrated in adversarial domains, where success is defined as outperforming an opponent 25 , collaboration requires agents to adapt to partners whose behaviors are diverse, unpredictable, and influenced by individual preferences and competency levels 13 . In real-world collaborative scenarios, human partners may demonstrate diverse coordination styles and skill levels, requiring AI agents to rapidly adjust their behavior to maintain effective collaboration. Early approaches relied on behavior cloning (BC) from human demonstrations 4 , but these methods are costly and struggle to generalize beyond the training data. Deep hierarchical reinforcement learning (DHRL) 22 provides a promising alternative by decomposing tasks into reusable high-level skills and low-level actions. These skills capture common patterns of cooperative behavior and enable agents to adapt more efficiently in complex environments 6 ; 16 . Despite these advances, existing DHRL skill discovery methods face key limitations in human–AI collaboration. Some approaches 6 ; 10 often produce redundant skills that capture noisy variations in observations rather than meaningful partner-relevant behavior. Other methods 16 are prone to skill collapse, where multiple skills converge to similar behaviors, reducing diversity and adaptability. As a result, agents may overfit to the partners encountered during training and fail to generalize to novel partners with different skill levels or behavioral styles, limiting the effectiveness of hierarchical policies in multi-agent coordination tasks. To address these limitations, we propose Intrinsic Action Disentanglement (IAD), a bi-level deep hierarchical reinforcement learning framework for training adaptive, partner-aware collaborative agents. The high-level skill manager observes temporally extended patterns of partner behavior and determines a skill that effectively guides the interaction. The low-level policy executes sequences of atomic actions conditioned on the selected skill. A key contribution of IAD is a novel intrinsic reward applied to the low-level policy, which encourages each skill to produce a distinct and predictable action sequence. This reward aligns the agent’s responses with partner behavioral dynamics, such as coordination style or skill level, ensuring that each skill captures a coherent mode of interaction. By disentangling skills in this way, the low-level policy learns diverse and structured action patterns, while the high-level manager dynamically selects skills that best respond to the partner’s temporally extended behavior. This bi-level optimization yields a reusable skill set that generalizes robustly to previously unseen partners. Our main contributions are summarized as follows: • We propose IAD, a novel bi-level DHRL framework for human-AI collaboration. IAD enables agents to adapt effectively to previously unseen partners characterized by diverse coordination styles and skill levels, resulting in robust coordination in collaborative tasks. • We introduce a novel intrinsic reward for the low-level policy that explicitly encourages disentanglement of atomic action sequences across skills. This reward aligns each high-level skill with a distinct, partner-relevant behavioral pattern, preventing skill collapse and producing reusable, skill-conditioned actions that facilitate effective high-level decision-making. • We conduct extensive evaluations in the Overcooked-AI environment across multiple layouts with diverse coordination challenges. Our experiments test generalization to previously unseen partners, including (i) large self-play partner populations with varying skill levels and coordination styles, (ii) a behavior-cloned human-proxy model trained from real human–human trajectories, and (iii) real human partners. We further analyze skill usage patterns to demonstrate that IAD produces diverse, specialized, and adaptive skill-conditioned behaviors that support effective coordination. 2 Related Work Developing agents that coordinate effectively with humans has been a major focus of recent research 4 ; 11 . Carroll et al. 4 introduced the Overcooked-AI environment and trained PPO agents alongside human proxy models derived from human–human gameplay trajectories. Although this enhances robustness to human interaction, it relies on extensive human data, which is expensive to obtain. Hao et al. 11 propose intrinsic rewards to encourage exploration of states that yield sparse extrinsic rewards when coordinating with human proxies. Fictitious Co-Play (FCP) 21 generates a pool of self-play policies and their historical versions, enabling adaptive agents without human data while capturing diverse partner behaviors. Extensions to FCP and other works further enhance heterogeneity using entropy-based objectives during training 17 ; 9 ; 28 ; 16 , and Hidden-utility Self-Play (HSP) 26 models human biases as hidden reward functions to train agents that cooperate with unseen humans. While these approaches improve partner diversity, they generally learn single-level policies, limiting their ability to reason over temporally extended behaviors critical for effective human–AI coordination. DHRL provides a principled framework for tasks requiring temporally extended behavior 22 ; 7 ; 18 . Classical DHRL methods, including options 2 ; 6 and feudal learning 23 , demonstrate the benefits of temporal abstraction in single-agent settings, with recent work extending these ideas to cooperative multi-agent and human–AI scenarios 16 ; 24 . Methods such as DIAYN 6 and related multi-agent variants 24 ; 10 encourage behavioral diversity via intrinsic rewards that maximize mutual information between skills and states or actions. While effective in continuous control tasks 27 , these approaches are less suited to multi-agent coordination 4 , as they promote diversity without explicitly modeling interaction dynamics or partner-dependent behavior. Moreover, they can be overly sensitive to minor, partner-irrelevant state variations, including those induced by the agent’s own actions, which may cause the high-level policy to treat trivial differences as distinct skills. As a result, the learned skills may fail to capture meaningful, partner-relevant behaviors, limiting adaptability and generalization in human–AI collaboration. Hierarchical Population Training (HiPT) 16 improves skill diversity by introducing an influence reward for shaping the high-level policy reward. However, the low-level policy in HiPT is trained solely with extrinsic sparse rewards and does not explicitly encode partner-dependent interaction structure, which can lead to skill collapse and limit adaptive coordination. Furthermore, both HiPT and HiPPO 15 randomly terminate high-level skills without proper termination criteria for low-level policies, which can limit efficient adaptation to dynamically changing partner behaviors. Overall, these limitations indicate that existing approaches either promote diversity without modeling interaction dynamics or rely on extrinsic rewards that fail to capture partner-specific coordination patterns. This underscores the need for hierarchical policies with interaction-aware intrinsic objectives. In contrast, our method introduces

Read the complete text