A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment

Koji Inoue, Yuki Okafuji, Jun Baba, Yoshiki Ohira, Katsuya Hyodo, Tatsuya Kawahara · 2025

Read A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment by Koji Inoue, Yuki Okafuji, Jun Baba, Yoshiki Ohira, Katsuya Hyodo, Tatsuya Kawahara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06241.txt.

Introduction

Koji Inoue and co-authors test a noise-robust turn-taking system for a dialogue robot in a real shopping mall, in a paper from 2025. Knowing when to speak and when to listen shapes how natural a robot chat feels. Lab tests of turn-taking models say little about noisy public spaces.

Their system predicts voice activity with a Transformer model built to shrug off noise. In the mall trial it was compared with a standard cloud speech service. The new system cut response delay clearly. Both the robot and its users answered faster, making the exchange feel more natural. Users also rated the faster system as a better experience. The authors argue that speed of response is a core part of chat quality.

Catalog record

CREATORKoji Inoue, Yuki Okafuji, Jun Baba, Yoshiki Ohira, Katsuya Hyodo, Tatsuya Kawahara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06241

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment Koji Inoue, Yuki Okafuji, Jun Baba, Yoshiki Ohira, Katsuya Hyodo, Tatsuya Kawahara Abstract Turn-taking is a crucial aspect of human-robot interaction, directly influencing conversational fluidity and user engagement. While previous research has explored turn-taking models in controlled environments, their robustness in real-world settings remains underexplored. In this study, we propose a noise-robust voice activity projection (VAP) model, based on a Transformer architecture, to enhance real-time turn-taking in dialogue robots. To evaluate the effectiveness of the proposed system, we conducted a field experiment in a shopping mall, comparing the VAP system with a conventional cloud-based speech recognition system. Our analysis covered both subjective user evaluations and objective behavioral analysis. The results showed that the proposed system significantly reduced response latency, leading to a more natural conversation where both the robot and users responded faster. The subjective evaluations suggested that faster responses contribute to a better interaction experience. I Introduction In recent years, large language models (LLMs) have advanced rapidly, driving the development of chatbots and spoken dialogue systems capable of facilitating natural human-machine interactions. By leveraging LLMs, dialogue robots have significantly improved language processing accuracy and response diversity, leading to an increasing number of real-world applications. In particular, cloud-based speech recognition and response generation services offer high versatility and ease of system updates and extensions. However, real-world deployment also presents challenges, such as network latency and environmental noise, which cannot always be effectively mitigated. These factors create significant obstacles to maintaining smooth human-robot interactions in practical settings. Turn-taking, a fundamental mechanism that determines when a system should start and stop speaking, is one of the most critical functions for ensuring fluid and natural interactions between humans and robots [ 1 ] . However, in real-world environments such as shopping malls and public spaces, where background noise and interruptions are common, conventional speech-recognition-based end-of-utterance detection frequently suffers from misrecognition and delays, leading to turn-taking breakdowns. As a result, users may interrupt due to impatience, or the robot may respond at inappropriate moments, causing disengagement and diminishing the overall user experience [ 2 , 3 ] . This study applies a voice activity projection (VAP) model, based on the Transformer architecture [ 4 ] , to enhance turn-taking in dialogue robots. VAP is a predictive model that estimates near-future voice activity for both conversational participants based on their audio input. By leveraging these predictions, the robot can regulate its turn-taking behavior more effectively. The key features of this model include: (1) End-to-end architecture: Directly predicting turn-taking from audio waveforms, making it robust against speech recognition errors. (2) Continuous, full-duplex processing: Allowing turn-taking prediction even while the user is still speaking. (3) Lightweight design: Enabling real-time processing in local environments, even without a GPU [ 5 ] . While prior research has implemented a VAP model in a dialogue robot [ 6 ] , no studies have focused on training a noise-robust VAP model and validating its performance through real-world field experiments. In this study, we enhance the VAP model by training it on simulated multi-condition data, including noisy environments, to improve turn-taking prediction accuracy in practical settings. To evaluate the effectiveness of the proposed system, we deployed the robot in a shopping mall and conducted a comparative experiment against a conventional cloud-based speech-to-text system. The evaluation included subjective user assessments (e.g., smoothness and satisfaction) alongside objective behavioral analyses, such as system and user reaction times. By examining these factors from multiple perspectives, we aimed to uncover qualitative improvements in interaction quality resulting from the enhanced turn-taking system. Conducting such a detailed subjective-objective behavioral analysis in real-world field experiments represents a novel approach that has rarely been explored in previous studies. The contributions of this study are as follows: • We developed a noise-robust VAP model by training it on multi-condition data simulating real-world environments and integrated it into a dialogue robot. • We conducted a real-world field experiment in a public shopping mall, evaluating the robot’s performance through both subjective user assessments and objective, quantitative behavioral analyses. II Related work Traditional spoken dialogue systems typically determine turn-taking using end-of-utterance detection based on silence or explicit cues, such as pressing a button. A common heuristic is to wait for a fixed period of silence (e.g., 500 ms or 1 s) before the system takes its turn [ 1 ] . While these methods are straightforward and reduce turn-taking complexity, they often result in unnatural interactions. Silence-based approaches also struggle in noisy environments, where background noise can interfere with voice activity detection, leading to turn-taking errors. Recent advancements leverage Transformer-based models to predict turn shifts more accurately [ 7 , 8 , 9 ] . A recent notable innovation is the VAP model, which predicts future speech activity from audio inputs in an end-to-end manner at the frame-wise level [ 4 , 10 ] . In human-robot interaction (HRI), turn-taking is typically based on either silence detection or cloud-based speech recognition systems. Several efforts have been made to implement turn-taking models in dialogue robots [ 11 , 12 ] . Research indicates that robots equipped with the latest VAP technology can respond more quickly than conventional cloud-based turn-taking systems [ 6 ] . However, these systems have primarily been tested in controlled laboratory environments and have not yet demonstrated robust performance in real-world settings, such as shopping malls with ambient noise. Within HRI, researchers have extensively studied the effects of turn-taking, or response delay, on various robotic platforms. Most studies focus on subjective evaluations such as user preferences and satisfaction in closed environments such as laboratories [ 13 , 14 , 15 , 16 , 17 , 18 ] . On the other hand, while some studies have observed how delays in robot responses affect user behavior in real-world environments [ 19 , 20 ] , research specifically investigating how response speed influences user behavior change in practical settings remains relatively unexplored. III A Noise-robust Turn-taking Model This section describes a noise-robust turn-taking model implemented in our robot system used in this study. We first provide an overview of the VAP model, followed by a newly trained VAP model designed for robustness in noisy real-world environments. III-A Voice Activity Projection Fig. 1 illustrates the architecture of the VAP model. The input consists of separated audio waveforms from two speakers, such as a user and a robot, each encoded using Contrastive Predictive Coding (CPC). The CPC encoder is pre-trained on 600,000 hours of data from the LibriSpeech dataset [ 21 ] , and its parameters remain frozen during VAP training. The encoded audio from each speaker is processed by a separate Transformer. These outputs are then passed through a cross-attention Transformer, allowing the model to attend to interactions between the two speakers. Finally, a linear layer specific to each task generates the output. Fig. 1: Architecture of VAP The original VAP model is designed for two tasks: voice activity projection (VAP), which predicts voice activity for the next two seconds, and voice activity detection (VAD), which detects ongoing voice activity. For the VAP task, the upcoming two seconds are divided into four time bins [ 4 ] , as shown in Fig. 2 . Since

Read the complete text