SRWToolkit: An Open Source Wizard of Oz Toolkit to Create Social Robotic Avatars

Atikkhan Faridkhan Nilgar, Kristof Van Laerhoven, Ayub Kinoti · 2025

Read SRWToolkit: An Open Source Wizard of Oz Toolkit to Create Social Robotic Avatars by Atikkhan Faridkhan Nilgar, Kristof Van Laerhoven, Ayub Kinoti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04356.txt.

Introduction

Atikkhan Faridkhan Nilgar and two co-authors present an open-source toolkit for prototyping social robot characters in a 2025 research paper. The Wizard of Oz method lets a hidden human, or here a language model, drive a character while researchers test interactions. Their web toolkit, SRWToolkit, runs language models locally on the device instead of in the cloud. Visitors can talk by typing, pressing a speech button, or using a wake word, while a control panel adjusts appearance, behavior, language, and voice in real time.

Eleven participants built roles such as driving assistant, mathematics teacher, and hospital receptionist. Ratings for usability, trust, and experience came out positive. Human-robot interaction studies can spin up custom avatars quickly.

Catalog record

CREATORAtikkhan Faridkhan Nilgar, Kristof Van Laerhoven, Ayub Kinoti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-04356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SRWToolkit: An Open Source Wizard of Oz Toolkit to Create Social Robotic Avatars Atikkhan Faridkhan Nilgar, Kristof Van Laerhoven, Ayub Kinoti Abstract We present SRWToolkit, an open-source Wizard of Oz toolkit designed to facilitate the rapid prototyping of social robotic avatars powered by local large language models (LLMs). Our web-based toolkit enables multimodal interaction through text input, button-activated speech, and wake-word command. The toolkit offers real-time configuration of avatar appearance, behavior, language, and voice via an intuitive control panel. In contrast to prior works that rely on cloud-based LLMs services, SRWToolkit emphasizes modularity and ensures on-device functionality through local LLM inference. In our small-scale user study, [n = 11] participants created and interacted with diverse robotic roles (hospital receptionist, mathematics teacher, and driving assistant), which demonstrated positive outcomes in the toolkit’s usability, trust, and user experience. The toolkit enables rapid and efficient development of robot characters customized to researchers’ needs, supporting scalable research in human-robot interaction. 1 Introduction and Related Work Social robotics research often encounters significant barriers due to the complexity and multidisciplinary nature of building interactive and engaging robot systems from scratch. These challenges require expertise in robotics, artificial intelligence (AI), and human-computer interaction design, limiting rapid experimentation and prototyping [ 9 , 7 , 5 , 2 ] . Traditional development approaches can be cumbersome, costly, and time-consuming, preventing greater participation from researchers who lack extensive technical backgrounds [ 6 ] . Current advances in large language models (LLMs), specifically GPT-4, have demonstrated impressive conversational capabilities, significantly enhancing natural language understanding and generation [ 2 ] . By embedding these models within a modular Wizard of Oz (WoZ) framework, researchers can rapidly prototype sophisticated human-robot interaction (HRI) scenarios while significantly reducing technical complexity. Prior studies have extensively explored WoZ methodologies as a way to rapidly prototype interactive systems without extensive programming. For instance, WoZ4U [ 6 ] is an open-source web-based interface designed to facilitate human-robot interaction studies by allowing experimenters to control robot behaviors in real-time with minimal coding effort. However, WoZ4U primarily addresses manual wizard control and does not integrate automated conversational agents. More recently, Fang et al. [ 2 ] explored “LLM Wizards”, where LLMs themselves act as automated wizards within WoZ setups. This approach showed a significant reduction in manual workload while enabling richer, more scalable interactions. Such integration allows researchers to benefit from the conversational skills of contemporary LLMs, creating natural, contextually relevant interactions without extensive manual scripting or training. Similarly, the WebWOZ [ 7 ] platform offers a generic architecture aimed specifically at language technology research, providing modular, scalable support for speech and text modalities. Popular LuxAI’s QTrobot ( https://luxai.com ) also integrates GPT-based dialogue to support learning in educational and therapeutic contexts. However, most of these WoZ systems rely on cloud-based LLM inference, which raises concerns related to data privacy, user consent regarding data handling and storage, as well as increased latency in real-time interactions. We address these challenges by presenting a modular WoZ toolkit that integrates local LLMs to enable the rapid development and deployment of social robotic avatars. These avatars refer to virtual representations of physical social robots, typically displayed on screens. Although they lack physical presence, they often retain similar interaction behaviors and characteristics, making them useful in scenarios where physical embodiment is not required. Our toolkit builds on prior works’ insights, combining the strengths of WoZ4U’s web-based modularity and LLM Wizards’ automation capabilities. We provide a web-based module toolkit that supports multimodal interaction, secure local LLM execution, and comprehensive session-based log-in to facilitate adaptive interaction. 2 Technical Architecture The social robotic avatar toolkit consists of three interconnected layers: the frontend, the backend, and the LLMs, which are detailed below. Frontend Implementation. The frontend is implemented as a browser-based application using JavaScript. The development environment utilizes node.js (v20.15.0) and yarn (v1.22.22) as the package manager. The frontend includes two main interfaces: the Control Panel (Figure 2 ) and the Social Robot Screen (Figure 2 ). The control panel is developed using React.js ( https://react.dev ) and Redux ( https://redux.js.org ). It allows administrators to initiate sessions with human-readable communication IDs, configure avatar visuals, select communication languages, enable or disable interaction modes, choose LLM models, write custom prompts (LLM prompts to create a role/character of the robot), and define the voice gender. These configurations are sent in real time to the backend and stored in MongoDB ( https://mongodb.com ). WebSocket ( https://dev.mozilla.org/en-US/docs/Web/API/WebSockets_API ) ensures status updates, including robot state and connectivity, which are sent to the control panel every five seconds. The social robot screen is created using React.js to deliver a dynamic and interactive interface for end-users. Figure 1 : Control Panel allows administrators to manage configurations. Figure 2 : Social Robot Screen for users to interact via text and voice inputs. The social robot screen facilitates user interaction through text input, voice input activated by a button, and voice commands triggered by a wake word (“Hey Bot”). Voice inputs are recorded using the MediaRecorder ( https://dev.mozilla.org/en-US/docs/Web/API/MediaRecorder ), while the proactive interaction mode uses the SpeechRecognition ( https://dev.mozilla.org/en-US/docs/Web/API/SpeechRecognition ) to listen for the phrase of the wake word. Redux is used to manage dynamic avatar states such as animations, blinking, and response indicators. Inputs, such as text or base64-encoded audio, are transmitted via persistent WebSocket connections to the backend. Audio responses from the backend are decoded and played using the browser’s audio. Backend Implementation. The backend is implemented in Python using FastAPI ( https://fastapi.tiangolo.com ). It serves as the central processing unit, which handles input processing, configuration management, data storage, and communication with the LLMs. It operates on-server using Uvicorn ( https://www.uvicorn.org ) to support asynchronous interactions. Dependencies are managed via Poetry ( https://python-poetry.org ). The backend handles WebSocket connections with both the social robot and control panel interfaces and parses incoming data. Voice inputs are transcribed into text using the Google Cloud ( https://cloud.google.com ) Speech-to-Text API. The transcribed or original text and the user’s conversation history are forwarded to the LLM for context-aware response generation. The response is converted to speech using the Google Cloud Text-to-Speech API and sent back to the frontend. Data storage is managed using MongoDB, containerized with Docker ( https://docker.com ). The database collects ‘communications’ for session metadata and configurations, ‘chat_messages’ for user input, custom prompts, and LLM response logs. All deployment is restricted to a local network environment to maintain security. The LLM server does not store any audio or conversational content; it retains only anonymized metadata. LLMs Implementation. We used Ollama ( https://ollama.com ), a lightweight framework for deploying and serving large-scale language models via API on the local device (macOS, M2 Pro, Sequoia 15.0.1, 32 GB). We included Llama 3.2, Gemma 2.0, Phi 3.5, Qwen 2.5, and Nemotron-mini 1.0, all of which are pulled and served through Ollama’s API. The LLM API is configured via a one-step port setup that links the toolkit to the GPU of the local device where Ollama runs. This ensures a secure, high-performance, and low-latency environment for model inference. The backend sends HTTP POST requests to the LLM API containing user queries and conversation context. The LLM processes these inputs and returns JSON-formatted responses. None of the data is retained by Ollama, ensuring

Read the complete text