Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction
Sam O’Connor Russell, Naomi Harte · 2025
Read Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction by Sam O’Connor Russell, Naomi Harte (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21043.txt.
Introduction
Sam O'Connor Russell and Naomi Harte show, in a 2025 paper, that adding visual cues improves computer prediction of turn taking in two person conversation. Smooth alternation is what makes talking with a robot or video partner feel natural. Most prediction models listen only to speech, yet human turn taking is richly multimodal.
Their model, MM-VAP, combines voice with facial expression, head pose, and gaze. On videoconference interactions it predicted holds versus shifts at 84 percent accuracy, against 79 for the best audio only rival. Breaking results down by the length of silence between turns showed gains at every duration. Ablation tests found facial expression contributed the most. The authors also checked that automatic speech alignment can train such models from telephone speech. Their working claim: when partners can see each other, a model needs eyes as well as ears.
Catalog record
| CREATOR | Sam O’Connor Russell, Naomi Harte |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21043
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction Sam O’Connor Russell, Naomi Harte Abstract Turn-taking is richly multimodal. Predictive turn-taking models (PTTMs) facilitate naturalistic human-robot interaction, yet most rely solely on speech. We introduce MM-VAP, a multimodal PTTM which combines speech with visual cues including facial expression, head pose and gaze. We find that it outperforms the state-of-the-art audio-only in videoconferencing interactions (84% vs. 79% hold/shift prediction accuracy). Unlike prior work which aggregates all holds and shifts, we group by duration of silence between turns. This reveals that through the inclusion of visual features, MM-VAP outperforms a state-of-the-art audio-only turn-taking model across all durations of speaker transitions. We conduct a detailed ablation study, which reveals that facial expression features contribute the most to model performance. Thus, our working hypothesis is that when interlocutors can see one another, visual cues are vital for turn-taking and must therefore be included for accurate turn-taking prediction. We additionally validate the suitability of automatic speech alignment for PTTM training using telephone speech. This work represents the first comprehensive analysis of multimodal PTTMs. We discuss implications for future work and make all code publicly available. 1 Introduction There is 200 ms of silence on average between speaking turns in a two-party interaction Stivers et al. 2009 ; Levinson and Torreira 2015 , yet language formation takes at least 600 ms Indefrey 2011 . Turn-taking is therefore predictive : listeners plan their next turn while the speaker is still speaking Levinson and Torreira 2015 ; Garrod and Pickering 2015 . Multimodal cues including syntax, prosody and gaze support this process Holler et al. 2016 , enabling speakers to hold the floor or to shift to another speaker Skantze 2021 . An important goal of human-robot interaction (HRI) is to develop machines capable of real-time conversation Marge et al. 2022 . Turn-taking models are central to this objective. Current turn-taking models in consumer systems are reactive , deciding whether to speak after a human stops speaking. This results in dialogue that is less spontaneous than human interaction Li et al. 2022 ; Woodruff and Aoki 2003 . Predictive turn-taking models (PTTMs) aim to overcome these issues Skantze 2021 . Inspired by human turn-taking, PTTMs are neural networks trained to continually make turn-taking predictions, e.g. the probability of an upcoming shift Skantze 2017 . Most work on PTTMs has been conducted using corpora of two-party human interaction Skantze 2021 . Almost all PTTMs rely on speech to make predictions; visual information, such as facial expression, is ignored. This may suffice for PTTMs trained on telephone speech Skantze 2017 ; Li et al. 2022 , but when interlocutors can see one another, might visual cues be useful predictors? The psycholinguistics literature strongly suggests so. Barkhuysen et al. 2008 cropped short segments from recordings of questions and asked participants if they belonged to the middle or the end of the question. They were more accurate when presented with both audio and video than in audio-only and video-only conditions. In a recent study Nota et al. 2023 found that listeners were faster at recognising questions containing eyebrow frowns. Such studies underline the importance of visual cues in turn-taking and multimodal interaction more generally Holler and Levinson 2019 . Aims Despite the essential role visual cues play in turn-taking, state-of-the-art PTTMs rely only on speech. It is therefore unknown whether visual cues can improve PTTM performance. We therefore consider the following research questions: 1. Can visual cues improve the performance of predictive turn-taking models? 2. If so, which aspects of turn-taking benefit most from the inclusion of visual cues? Overview We introduce multimodal VAP (MM-VAP), a transformer-based PTTM which combines speech with visual features, including facial expression. We show our model outperforms the voice activity projection (VAP) model, a state-of-the-art audio-only PTTM Ekstedt and Skantze 2022b . PTTMs are typically trained using accurate phonetic alignments Ekstedt and Skantze 2022b ; Li et al. 2022 . We use automatic speech recognition (ASR) to reflect real-word conditions. We re-implement and re-train the VAP model on the Switchboard corpus of telephone speech Godfrey et al. 1992 . We find performance falls slightly due to automatic alignment errors, but the impact is minimal. Next, we train the VAP model using audio from the Candor corpus of videoconferencing interaction Reece et al. 2023 , again using ASR. We find good performance in distinguishing shifts from holds . We conduct a facial action unit analysis, which reveals that the next speaker displays enhanced mouth, lip, jaw and chin movements before speaking (i.e. before a shift). These are visual cues which multimodal PTTM could exploit. We therefore combine speech with facial expression, gaze, and head pose in our new multimodal PTTM and demonstrate superior performance over the VAP model (79% vs 83% balanced accuracy for hold/shift prediction). The performance increase is most notable in the F_{1} score for shifts, with a 6-10% increase in the F_{1} score. In prior work, PTTM performance is reported as a single figure for all holds and shifts. In a more comprehensive analysis, we group holds and shifts by duration of silence between turns. This reveals the strength of our multimodal model across all durations of silence between turns (gaps) and overlapping speech. The performance of both MM-VAP and VAP is worse when there are longer gaps in the Candor corpus, however, MM-VAP outperforms the audio-only model across all durations of gaps considered (83% vs 79% for all gaps >0 ms, 78% vs 75% for all gaps >750 ms). We discuss our working hypothesis that interlocutors utilise visual cues when they can see one another , and they must therefore be included to maintain robust performance. Finally, we conduct a detailed ablation study. This reveals that facial action units, which encode facial expression, are the biggest contributors to the increased accuracy over the audio-only turn-taking model. We conclude with a discussion of our findings, highlighting the vital importance of non-verbal communication, which researchers in human-robot interaction must consider going forward. We make our code publicly available for future research 1 1 1 https://github.com/russelsa/mm-vap . 2 Background Turn-taking In a conversation, the current speaker either holds the turn or shifts to another interlocutor Sacks et al. 1974 . The time between turns is the floor-transfer offset (FTO), which is positive for a gap and negative for an overlap Heldner and Edlund 2010 . A typical two-party interaction has a mean FTO of +200 ms Stivers et al. 2009 . There is no single definition of a turn. In Conversation Analysis, a turn is defined by social action (a question, agreement, etc.) Kasper and Wagner 2014 . In predictive turn-taking, a turn is identified from voice activity (VA), which indicates the presence or absence of speech at any moment in time Ekstedt and Skantze 2022b ; Li et al. 2022 . Turn-taking is aided by multimodal cues. The words that we speak as well as how we speak (prosody) play important roles. Bögels and Torreira 2015 found prosody is essential for listeners to correctly determine the end of the turns with multiple completion points e.g. " are you a student \backslash here \backslash at this university \backslash .". Visual cues include gaze. Speakers look away at