M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal · 2026

Read M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction by Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11975.txt.

Introduction

Hasan, Lee, Sarker, and Iqbal introduce M2HRI, a multi-mode multi-agent setup that treats each robot as an identity-bearing agent with personality and long-term memory, plus coordination guiding participation. Old multi-robot systems often treat robots as the same, missing how distinct identities shape user view and coordination needs. In a study with 105 people, most personality contrasts proved distinct and always shown. Long-term memory raised preference awareness and natural interaction, while contextualised coordination raised chat flow, response fit, and overlap avoidance.

Catalog record

CREATORShaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40955
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11975

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal Abstract Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot works treat robots as functionally identical, overlooking how robots individual identity shape user perception and how coordination shapes multi-robot behavior when such individuality is present. To address this, we introduce M2HRI, a multimodal multi-agent framework built on large language models that equips each robot with distinct personality and long-term memory, alongside a coordination mechanism conditioned on these differences. In a controlled user study ( n = 105 n=105 ) in a multi-agent human–robot interaction (HRI) scenario, we find that LLM-driven personality traits are significantly distinguishable and enhance interaction quality, long-term memory improves personalization and preference awareness, and centralized coordination significantly reduces overlap while improving overall interaction quality. Together, these results demonstrate that both agent individuality and structured coordination are essential for coherent and socially appropriate multi-agent HRI. Project website and code are available at https://project-m2hri.github.io/ . I Introduction Human-Robot Interaction (HRI) has advanced considerably over the past decades, driven by the deployment of socially capable robots in everyday environments such as homes, hospitals, and educational settings [ 20 , 14 ] . Traditionally, HRI has focused on single-agent scenarios, where one robot interacts with a human to enable intuitive communication, effective task execution, and user engagement [ 12 , 26 ] . These early systems established fundamental interaction mechanisms, including turn-taking, intent recognition, and affective response, demonstrating that robots could function as effective social partners, particularly in structured environments. However, as real-world applications have grown in complexity and scale, the limitations of single-agent HRI have become evident. A single robot is often insufficient to simultaneously attend to multiple users, operate across shared spaces, or support the diversity of roles required in more complex social environments [ 28 ] . This has led to a growing interest in multi-agent HRI, where multiple robots interact with a single human or group of users. In such settings, interaction is no longer centered on an individual agent, but instead emerges from the collective behavior of a team of robots. Consequently, the problem shifts from designing isolated interactions to orchestrating coherent group behavior. This transition introduces new challenges, including managing conversational turn-taking, assigning roles among agents, and maintaining a consistent and socially appropriate group identity [ 21 , 9 ] . Figure 1: Multimodal multi-agent human-robot interaction scenario. A human user interacts with two NAO robots, each with distinct personality, memory, and perception, enabling personalized and context-aware embodied interaction. Recent advances in vision-language models (VLMs) and large language models (LLMs) have enabled modern HRI systems to leverage multimodal perception, allowing robots to jointly reason over speech and visual observations to generate contextually appropriate responses [ 16 ] . However, multimodal multi-agent HRI introduces significant challenges in coordinating multiple robots and maintaining effective interaction. Prior work has explored centralized coordination strategies using shared context and perceptual cues to determine turn-taking and response selection [ 34 ] . Research on robot-robot dynamics further shows that inter-agent behaviors, including collaboration and information exchange, can strongly influence user perception, trust, and engagement [ 5 , 13 ] , while studies on interaction transitions emphasize the need for communication to maintain continuity and user understanding [ 30 ] . Additional work highlights how motion, spatial behavior, embodiment, and collective dynamics shape user attention, cognition, and emotional response [ 19 ] . Multi-robot systems have also been investigated in application-driven contexts such as collaborative creativity and complex workflows, demonstrating both their potential and the coordination challenges they introduce [ 25 , 7 ] . Despite this progress, a critical gap remains in how agent individuality are modeled in multi-robot HRI scenario. Existing works in multi-robot HRI treat agents as functionally interchangeable, overlooking personality and interaction history, yet in shared social environments, socially appropriate behavior is inherently personal. A robot that maintains a consistent personality and remembers user preferences provides a markedly different experience than one that treats each interaction independently, and this distinction becomes even more critical in multi-robot HRI settings, where individuality determines whether the system is perceived as distinct social agents or identical machines acting in parallel. Furthermore, introducing such heterogeneity makes coordination more challenging: deciding which robot should respond can no longer rely on simple turn-taking rules, but must instead account for differences in identity to ensure coherent group behavior. To address this gap, we investigate three research questions: • RQ1: How does robot personality affect agent distinguishability and interaction engagement in multi-agent HRI? • RQ2: How does robot’s long-term memory influence personalization and contextual awareness in multi-robot interaction? • RQ3: How does coordination among robot agents affect interaction quality? To address these research questions, we design and implement M2HRI, a multimodal multi-agent HRI framework built on LLMs and VLMs that integrates perception, memory, personality, and centralized coordination within a unified system (Fig. 1 ). We evaluate the framework through a controlled human-subject study ( n=105 ), varying agent personality, memory, and coordination. Our results show that personality enables users to reliably distinguish between agents, recognize intended traits, and enhances interaction engagement, though its effectiveness varies with how behaviorally expressive a trait is. Memory significantly improves recall accuracy, preference awareness, and perceived personalization, while coordination plays a critical role in ensuring coherent interaction by improving conversational flow, response appropriateness, and reducing overlap. Together, these findings indicate that modeling agent individuality and coordination are both necessary and complementary for coherent, personalized, and socially appropriate multi-robot interaction. II Related Work II-A Multi-Agent HRI Research in multi-agent HRI shows that interacting with multiple robots introduces a fundamentally different and more complex social context than single-robot settings, shaping human cognition, behavior, and emotion [ 27 , 19 ] . Prior work demonstrates that the mere presence, number, and coordination patterns of multiple robots can significantly influence task performance, feelings of being observed, and emotional responses, highlighting the role of embodiment and collective motion in interaction outcomes [ 19 ] . Beyond presence effects, multi-robot interactions give rise to rich social dynamics such as conformity, compliance, and group perception, where humans adapt their behavior based on robot group behavior and structure [ 27 , 32 ] . In particular, studies on robot–robot relationships show that ingroup and outgroup dynamics among robots influence human compliance, trust, and collaboration, altering how people respond to instructions and perceive robotic teams [ 32 ] . At the system level, recent work highlights emerging challenges in orchestration, coordination, and interaction management in multi-agent systems, while practical HRI studies reveal difficulties in controlling and designing interactions involving multiple robots [ 11 , 6 ] ; however, existing approaches primarily focus on behavioral or system-level aspects, with limited attention to integrating agent-level attributes such as personality and memory for multi-agent interaction [ 8 ] . II-B Robot Personality and Memory in HRI Robot personality has been widely studied in single-agent HRI, with the Big Five Factor Model serving as the dominant framework for operationalizing distinct traits through speech, gesture, and language style, and studies finding that personality traits influence trust and engagement, and that individual user traits moderate these effects [ 18 , 22 ]

Read the complete text