Voice Activity Projection Model with Multimodal Encoders

Takeshi, Catherine · 2025

Read Voice Activity Projection Model with Multimodal Encoders by Takeshi, Catherine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03980.txt.

Introduction

Takeshi Saga and Catherine Pelachaud improve turn taking prediction for talking machines with sight and sound together. Their paper from 2025 deals with voice activity projection, which forecasts who will speak next from more than silence gaps. Social talk is multimodal, so face cues can carry what voice misses.

Pre trained audio and face encoders feed their model, catching subtle signs like gaze and mouth shifts. On turn taking measures the model matched the best past systems and beat them in some cases. Code and trained models are public. The work pushes machine talk partners toward smoother, more human timing in conversation.

Catalog record

CREATORTakeshi, Catherine
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Voice Activity Projection Model with Multimodal Encoders Takeshi, Catherine Abstract Turn-taking management is crucial for any social interaction. Still, it is challenging to model human-machine interaction due to the complexity of the social context and its multimodal nature. Unlike conventional systems based on silence duration, previous existing voice activity projection (VAP) models successfully utilized a unified representation of turn-taking behaviors as prediction targets, which improved turn-taking prediction performance. Recently, a multimodal VAP model outperformed the previous state-of-the-art model by a significant margin. In this paper, we propose a multimodal model enhanced with pre-trained audio and face encoders to improve performance by capturing subtle expressions. Our model performed competitively, and in some cases, even better than state-of-the-art models on turn-taking metrics. All the source codes and pretrained models are available at https://github.com/sagatake/VAPwithAudioFaceEncoders . Saga Pelachaud Sorbonne UniversityFrance CNRSFrance keywords turn taking, voice activity projection, multimodal modeling † † email: [email protected], [email protected] 1 Introduction Turn-taking is a fundamental and universal function in social interactions, enabling the coordinated exchange of speaking turns between participants in a conversation, which is essential for successful social communication [ 1 , 2 ] . Smooth turn-taking enables coherent dialogue, preventing everyone from speaking simultaneously. We can divide turn-taking cues into three aspects: turn-yielding cues, turn-holding cues, and turn-initial cues [ 3 ] . Turn-yielding cues (e.g., the falling pitch at the end of the sentence) indicate the speaker’s intentions to yield. In contrast, turn-holding cues (e.g., level intonation) show that the speaker wants to continue their turn. Unlike these cues, turn-initial cues (e.g., raising an index finger) can be produced by the following speaker candidates to explicitly indicate that they wish to speak next or, in some cases, even interrupt the current speaker’s speech. These cues are multimodal. In particular, the cues include text (e.g., syntax, pragmatics, semantics, fillers) [ 4 ] , speech (e.g., pitch, intensity, formants) [ 5 ] , pauses (filled pause, unfilled pause) [ 6 ] , hand/arm gestures (termination, tense, rhythmic) [ 3 ] , head movement (nodding, shaking, tilting) [ 7 ] , postures (leaning, swelling) [ 8 ] , and so on. Interlocutors produce and rely on these cues to take, yield, or hold turns, where reproduction is challenging in robots or embodied conversational agents. In addition to this multimodal understanding, turn-taking timing is another crucial factor for smooth dialogue, especially in sustained interactions between humans and robots or humans and agents [ 9 ] . Participants perceived agents with too early responses (interruptions) as less agreeable and more assertive [ 10 , 11 ] . In contrast, agents with longer pauses give the impression of creating more rapport [ 12 , 10 ] . Although human language production systems require about 600 ms to produce the first response word, the average response time from the end of the preceding utterance is about 200 ms [ 13 ] . This implies that humans may use a predictive approach to content or timing. As Meyer mentioned, sentence processing in the human brain is highly incremental and predictive, with evidence that speakers can prepare utterances while listening to another person’s speech [ 14 ] . For example, speakers can rapidly grasp the content and speech acts. They can also predict the ends of turns quickly and precisely. Several turn-taking management models have been proposed to simulate this incremental and predictive sentence processing [ 15 , 16 ] . Turn-taking management is vital for sustained interaction between humans and artificial agents [ 9 ] . Participants perceived agents with too early responses (interruptions) as less agreeable and more assertive [ 10 , 11 ] . In contrast, agents with longer pauses give the impression of creating more rapport [ 12 ] . ERICA by Lala et al. is an iconic example of automatic human-robot turn-taking [ 17 ] . The authors used logistic regression models separately for predicting backchannel generation and turn-taking timing. They calculate prosodic features, such as pitch and intensity, over multiple time windows and predict the likelihood of occurrence within the next 500 milliseconds. Instead of predicting backchannels and turn-taking timing separately, Ekstedt proposed to predict them with a unified representation called voice activity projection state [ 18 ] . Using the unified representation, they successfully predicted both the timings of the turn shift and the back channel. However, this model relies only on audio signals. It cannot predict turn-taking timings that involve cues such as gaze direction, mouth opening, and other nonverbal gestures [ 19 , 20 , 21 ] . Recently, Onishi et al. proposed a model that takes as input nonverbal signals (facial action units corresponding to facial muscular contractions, head poses, and body joint positions) and audio signals [ 22 ] . Although the model improved prediction performance, these nonverbal features can represent only superficial characteristics, excluding context-rich information such as sequential, spatial, relational, and temporal relationships between them. Other researchers have proposed text-based LLM approaches, but they still have challenges running in an auto-regressive manner in nearly real-time [ 23 , 24 ] . More recently, though speech-to-speech foundation models such as Mini-Omni or Moshi (Gemini Live or ChatGPT voice chat as service-level examples) showed very natural speech conversation abilities, investigations on internal mechanisms of those fully end-to-end models are still ongoing research, which is generally very challenging in terms of explainability [ 25 , 26 , 27 , 28 ] . In this paper, we aim to thoroughly consider nonverbal turn-taking cues, proposing to enhance Ekstedt’s voice activity projection model with pre-trained encoders for audio and facial signals [ 18 ] . We hypothesize that combining the encoders will contribute to capturing the coordinated, multimodal intentions of the speakers, conveyed by both facial and audio signals. Specifically, by adding a facial image encoder, the subtle social context of facial expressions will be embedded into the network. We trained the multimodal prediction models to prove this idea and evaluated their performance. In the remainder of this paper, after presenting the variants of VAP models, we describe our model. 2 Voice activity projection One of the successful models in turn-taking prediction is the voice activity projection (VAP) by Ekstedt et al. [ 18 ] . Unlike conventional voice activity detection (VAD) models, which consider only the voice activity of a single speaker, the VAP model predicts both users’ speaking states (e.g., speaking or not speaking) and unified VAP states, aiming to model interactions better for turn-taking behavior prediction. Additionally, the VAP model employs a predictive approach, predicting what will happen next based on the previous audio signals. Figure 1 shows an example of the VAP state. VAP state is a discrete representation shaped as 2 × 4 binary bins. Each row corresponds to a time series of voice activity of each user. Based on the assumption that further activities are more challenging to predict, the duration of each time bin is defined as 0.2, 0.4, 0.6, and 0.8 sec. from the nearer future to the further future. In the training period, the model is trained by multi-task learning of future VAD prediction for each user and VAP state. This model has been validated in various aspects, including real-time prediction systems, the training effect on a multilingual

Read the complete text