A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot CCS: Computer systems organization Robotics CCS: Human-centered computing Natural language interfaces CCS: Software and its engineering Software development frameworks

Erich Studerus, Vivienne Jia Zhong, Stephan Vonschallen · 2026

Read A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot CCS: Computer systems organization Robotics CCS: Human-centered computing Natural language interfaces CCS: Software and its engineering Software development frameworks by Erich Studerus, Vivienne Jia Zhong, Stephan Vonschallen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21013.txt.

Introduction

Studerus and co-authors build an open framework that gives the Pepper robot fast, multimodal interaction with a language model. Many Pepper setups chain speech to text, a model, and text to speech, which adds delay and loses tone cues. Their framework uses end to end speech to speech models instead. Function calling lets the model plan robot actions, such as navigation, gaze, and tablet use. It also folds in vision, touch, and system state. The code runs on the robot's tablet or a plain Android device.

Catalog record

CREATORErich Studerus, Vivienne Jia Zhong, Stephan Vonschallen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19981
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot CCS: Computer systems organization Robotics CCS: Human-centered computing Natural language interfaces CCS: Software and its engineering Software development frameworks Erich Studerus, Vivienne Jia Zhong, Stephan Vonschallen Abstract . Despite recent advances in integrating Large Language Models (LLMs) into social robotics, two weaknesses persist. First, existing implementations on platforms like Pepper often rely on cascaded Speech-to-Text (STT) → \rightarrow LLM → \rightarrow Text-to-Speech (TTS) pipelines, resulting in high latency and the loss of paralinguistic information. Second, most implementations fail to fully leverage the LLM’s capabilities for multimodal perception and agentic control. We present an open-source Android framework for the Pepper robot that addresses these limitations through two key innovations. First, we integrate end-to-end Speech-to-Speech (S2S) models to achieve low-latency interaction while preserving paralinguistic cues and enabling adaptive intonation. Second, we implement extensive Function Calling capabilities that elevate the LLM to an agentic planner, orchestrating robot actions (navigation, gaze control, tablet interaction) and integrating diverse multimodal feedback (vision, touch, system state). The framework runs on the robot’s tablet but can also be built to run on regular Android smartphones or tablets, decoupling development from robot hardware. This work provides the HRI community with a practical, extensible platform for exploring advanced LLM-driven embodied interaction. 1. Background and Summary Recent advancements in Large Language Models (LLMs) have fundamentally transformed Human-Robot Interaction (HRI) ( 13 ) , enabling open-domain dialogue that replaces scripted systems with natural, context-aware conversation ( 21 ; 33 ) . Several recent studies have integrated LLMs into social robots like Pepper ( 2 ; 3 ; 4 ; 5 ; 6 ; 15 ; 21 ; 27 ) . While early implementations primarily used LLMs for dialogue generation, more recent work explores their potential for task planning, gesture generation, and multimodal behavior control ( 22 ; 23 ) . However, two major barriers prevent these implementations from fully realizing the LLM’s potential for multimodal, embodied interaction. First, most existing implementations rely on cascaded pipelines that sequentially process speech through separate Speech-to-Text (STT), LLM response generation, and Text-to-Speech (TTS) stages. This architecture introduces two critical problems: Each processing stage adds incremental delays, resulting in accumulated latency that prevents fluid real-time dialogue ( 7 ; 24 ) . Consequently, recent LLM implementations on platforms like the Pepper robot report system response times ranging from 3.84 to 8.96 seconds ( 10 ; 21 ) , far exceeding the 1–2 second threshold established as acceptable for natural conversation flow ( 11 ; 20 ; 26 ; 30 ) . Additionally, the conversion of audio to text and back to audio discards paralinguistic information, such as prosody, intonation, and emotional cues, that are crucial for natural, empathetic human-robot interaction ( 7 ; 21 ) . Second, despite recent progress, most implementations still fail to fully leverage the LLM’s capabilities for multimodal perception and agentic control. Achieving embodied AI requires the LLM to convert abstract natural language instructions into concrete, executable actions and integrate diverse sensor feedback in real-time ( 1 ; 12 ; 23 ) . This requires mechanisms like Function Calling (also known as “Tool Use”) ( 3 ; 22 ; 23 ) , where the LLM autonomously selects and invokes predefined functions with appropriate parameters. Through Function Calling, the LLM can orchestrate robot actions (navigation, gaze control, tablet interaction) and actively direct its perception by triggering targeted sensor data acquisition (e.g., camera images) ( 1 ; 25 ) . Additionally, the system must integrate asynchronous, event-driven inputs from haptic sensors ( 29 ) and device interfaces that provide contextual information about physical interactions and system state. The LLM’s native multimodal capabilities then process these diverse data streams to achieve embodied grounding in the physical environment ( 18 ; 19 ) . This submission presents an open-source Android framework for the Pepper robot that overcomes these barriers through two key innovations (see Figure 1 ): (1) Integration of end-to-end S2S models ( 18 ; 19 ; 9 ; 8 ) to achieve low-latency streaming interaction while preserving paralinguistic cues, and (2) extensive Function Calling capabilities that enable the LLM to orchestrate robot actions (navigation, gaze control, tablet interaction) and integrate diverse multimodal feedback (vision analysis, haptic feedback, system state). The framework provides a modular, easily deployable reference implementation that facilitates wider adoption of modern LLM features in embodied platforms. Its design is informed by insights from a prior field study investigating communication dynamics between older adults and an LLM-powered robot ( 34 ) . Figure 1. System architecture comparison showing traditional cascaded pipelines versus the proposed low-latency, multimodal framework. A diagram comparing traditional STT-LLM-TTS pipelines with the proposed framework using end-to-end S2S models. 2. Purpose The primary purpose of this framework is to provide the HRI community with a reusable, open-source framework that addresses the latency and limited agentic control challenges outlined above. It establishes a practical blueprint for using an LLM as an agentic orchestrator for multimodal robot behavior—an architectural approach that can be adapted to other robotic platforms. The framework is particularly valuable given the practical challenges facing Pepper researchers: following Aldebaran’s court-ordered liquidation on June 2, 2025 ( 14 ) , official support for Pepper has become uncertain, and the platform’s Android/Java ecosystem and aging hardware (Android 6.0, API level 23) ( 28 ) pose non-trivial integration barriers. Furthermore, a key purpose of this contribution is to simplify deployment for HRI research. The framework is designed to run both on the Pepper robot and on standard Android smartphones or tablets, decoupling the development cycle from physical robot hardware. This enables researchers to develop, test, and prototype interaction logic without requiring access to expensive robotic platforms. 3. Characteristics 3.1. Architectural Advantage Android-Native Execution: The entire application runs natively on Android devices—either on the Pepper robot’s onboard tablet (controlling the robot’s physical body via QiSDK) or on standard Android smartphones and tablets. This self-contained architecture eliminates the need for external computers or bridging software common in previous LLM-HRI integrations ( 2 ; 4 ; 5 ) , simplifying deployment and use. Dual Build-Flavor System: The framework can be deployed either on the Pepper robot or on standard Android smartphones and tablets through two build configurations. Robot-specific functions adapt automatically: The analyze_vision tool, for instance, accesses the robot’s camera system (via QiSDK) when running on Pepper, but uses the device’s front camera when running on a smartphone or tablet. Navigation and movement commands control the physical robot or are simulated accordingly. All other tools, including internet search, weather queries, games, and conversational features, operate identically regardless of the deployment platform. 3.2. Core Interaction Engine: Speech-to-Speech Models The framework’s low-latency capabilities are powered by end-to-end S2S models from multiple providers (OpenAI ( 18 ; 19 ) , Azure OpenAI, x.ai ( 32 ) , Google ( 9 ) ). A central TurnManager orchestrates the robot’s state (listening, thinking, speaking). Key advantages include: End-to-End Speech-to-Speech (S2S) Model: By processing audio directly, the system bypasses the cumulative latency of separate STT/TTS stages. Furthermore, unlike cascaded systems, where information is lost during text conversion, the S2S model can process paralinguistic cues like user intonation. Consequently, its generated speech is not merely text being read aloud; the model adapts

Read the complete text