MistyPilot: An Agentic Fast–Slow Thinking LLM Framework for Misty Social Robots
Xiao Wang, Lu Dong, Jingchen Sun, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju · 2026
Read MistyPilot: An Agentic Fast–Slow Thinking LLM Framework for Misty Social Robots by Xiao Wang, Lu Dong, Jingchen Sun, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03640.txt.
Introduction
Xiao Wang and five others turn a social robot into an autonomous assistant that non-programmers can direct. The Misty robot exposes open programming interfaces, but choosing tools, chaining them, and setting settings still defeats users without coding experience. Their 2026 approach, MistyPilot, wraps the robot in an agentic language model layer that selects, orchestrates, and configures tools from high-level instructions.
Two agents divide the labor. A physically interactive agent runs sensor-triggered, tool-driven tasks robustly. A socially intelligent agent makes dialogue aligned with the user's emotions. A fast-slow thinking scheme captures user preferences, cuts latency, and raises efficiency. The team contributes five benchmark data sets and tests routing correctness, task completeness, retrieval efficiency, tool scalability, and emotion alignment. Results support the approach across every dimension.
Catalog record
| CREATOR | Xiao Wang, Lu Dong, Jingchen Sun, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37313 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MistyPilot: An Agentic Fast–Slow Thinking LLM Framework for Misty Social Robots Xiao Wang, Lu Dong, Jingchen Sun, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju Abstract With the availability of open APIs in social robots, it has become easier to customize general-purpose tools to meet users’ needs. However, interpreting high-level user instructions, selecting and configuring appropriate tools, and executing them reliably remain challenging for users without programming experience. To address these challenges, we introduce MistyPilot, an agentic LLM-driven framework for autonomous tool selection, orchestration, and parameter configuration. MistyPilot comprises two core components: a Physically Interactive Agent (PIA) and a Socially Intelligent Agent (SIA) . The PIA enables robust sensor-triggered and tool-driven task execution, while the SIA generates socially intelligent and emotionally aligned dialogue. MistyPilot further integrates a fast–slow thinking paradigm to capture user preferences, reduce latency, and improve task efficiency. To comprehensively evaluate MistyPilot, we contribute five benchmark datasets. Extensive experiments demonstrate the effectiveness of our framework in routing correctness, task completeness, fast–slow thinking retrieval efficiency, tool scalability, and emotion alignment. All code, datasets, and experimental videos will be made publicly available on the project webpage. I INTRODUCTION Social robots, as emerging stars of the modern era, are increasingly involved in education, daily living, elder care, and emotional support, demonstrating advanced companionship and assistive capabilities [ 1 , 2 , 3 , 4 , 5 , 6 ] . Misty, a compact, intelligent, and elegantly designed social robot, is increasingly adopted for both interactive and assistive roles [ 7 , 8 , 9 ] . It offers atomic-level open APIs—including tactile sensing, articulated head and arm control, and camera and speech modules—that support the development of versatile functional tools such as sensor-triggered greetings, camera activation, content retrieval, conversational question answering, emotional support, and daily memos. However, high-level user instructions are often abstract, diverse, and inherently uncertain, making it impractical to enumerate and hard-code all possible scenarios. For example, a user might request a story about ’The Three Little Pigs’ and later wish to change the ending and hear it retold. A promising strategy to address such challenges is to pre-define a set of general, molecular-level tools that can be flexibly composed as needed. Much like possessing tools, leveraging them effectively presents its own challenges; Nevertheless, selecting and configuring appropriate tools remains a significant challenge for non-expert users. As user demands continue to evolve, scalability is essential for social robots. However, most existing systems lack plug-and-play functionality, requiring extensive manual coding and configuration to add or integrate tools. This rigid design limits adaptability to new tasks and diverse user needs. Therefore, next-generation social robots should enable seamless tool expansion, automatic detection of new tools, and efficient, accurate reasoning even as tool numbers increase. Large Language Model (LLM)-based tool agents hold great potential, yet research that integrates physical sensing, dialogue, and multimodal emotional alignment within a unified framework is scarce, particularly in the context of Misty social robots. Fig. 1: Overview of MistyPilot workflow for interpreting high-level human instructions. Instead of requiring professionals to hand-code robot behaviors and deploy features on the Misty robot, MistyPilot parses natural-language instructions, analyzes the task, selects and parameterizes tools from its library, and executes them on Misty Robot. Furthermore, most social robots continue to face significant challenges in achieving natural, emotionally attuned interactions. Many rely on basic text-to-speech (TTS) modules for voice generation and lack the ability to accurately perceive and align with emotional states, leading to responses that appear mechanical or emotionally detached. Moreover, these systems often fail to retain memory of prior interactions or adapt to user-specific preferences, resulting in repetitive reasoning and suboptimal interactions. This deficiency in emotional awareness and personalization greatly constrains their potential to serve as truly socially intelligent companions capable of fostering trust and sustaining long-term engagement. To address these challenges, we propose MistyPilot, as illustrated in Fig. 1 , an agentic LLM-driven pilot framework that enables autonomous tool orchestration and parameter configuration, empowering social robots to deliver natural, engaging interactions and positioning them as more effective assistants and companions. To the best of our knowledge, this is the first agentic framework that unifies physical sensing, dialogue generation, and emotional alignment to enable autonomous tool orchestration for social robots, supported by advanced reasoning and decision-making autonomy. Our main contributions are summarized as follows: • We present MistyPilot, an agentic LLM-driven framework for end-to-end interpretation of high-level user instructions, covering task analysis, tool selection, parameter configuration, and orchestration, while supporting plug-and-play tool expansion and maintaining system robustness. • We introduce a multi-agent architecture consisting of three components: a Task Router that serves as the central dispatcher, a Physically Interactive Agent (PIA) that handles dynamic sensor-triggered events and tool utilization, and a Socially Intelligent Agent (SIA) that integrates a fast-slow thinking script-writing paradigm to achieve fine-grained multimodal emotional alignment in dialogue. • We contribute five benchmark datasets and conduct extensive experiments on evaluating task routing correctness, PIA and SIA performance, fast–slow thinking retrieval, and tool extensibility, demonstrating MistyPilot’s effectiveness and robustness. Human evaluation further suggests that MistyPilot is preferred in terms of response naturalness, accuracy, emotional expressiveness, timeliness, and overall user satisfaction. Fig. 2: Overview of the MistyPilot framework. MistyPilot interprets high-level user instructions through a Task Router that dispatches tasks to either a Physically Interactive Agent (PIA) or a Socially Intelligent Agent (SIA) for automated tool utilization and parameter adaptation. The PIA oversees a Sensor & Tool Manager to dynamically orchestrate sensor-related and tool-dependent tasks, while the SIA maintains a Task Status Manager to track the current task state. Fast Thinking accelerates inference by memory hitting for stored knowledge and preferences, while Slow Thinking employs a Script Writer to interpret the task, align responses with fine-grained emotional expression, and deliver them via speaking and movement modules. II RELATED WORKS Recent advances in LLMs have demonstrated remarkable reasoning capabilities, motivating their integration into social robotics to enable end-to-end solutions for natural, high-level user interaction. II-A Tool-augmented LLMs Recent advances in tool-augmented LLMs reflect a shift from solely enhancing reasoning performance to pursuing broader and more comprehensive capabilities. Toolformer [ 10 ] introduces a self-supervised approach that enables LLMs to autonomously learn API calls and embed them directly into text generation, while ReAct [ 11 ] proposes interleaved reasoning and acting to dynamically query external tools and refine reasoning, effectively mitigating hallucination. ToolLLM [ 12 ] employs a tree-structured reasoning framework with rollback mechanisms to improve multi-tool orchestration, and Tulip Agent [ 13 ] proposes a vector store–based recursive tool retrieval system that scales to large tool libraries while reducing costs and supporting robotics applications. Benchmark datasets such as API-Bank [ 14 ] , APIBench [ 15 ] , and ToolAlpaca [ 16 ] have further advanced this research by enabling standardized evaluation and generalization testing. Building on these foundations, embodied and multimodal research such as AutoRT [ 17 ] , Odyssey [ 18 ] , and Speech-enabled LLM Robot Control [ 19 ] further advanced tool-augmented LLM tasks by integrating perception and planning in real-world scenarios. However, comprehensive tool orchestration for social robot remains largely underexplored, highlighting the motivation for MistyPilot. II-B Social Robots for Human–Robot Interaction Recent studies demonstrate the expanding role of social robots in providing companionship and emotional support across diverse