Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction

Hyunbae Jeon, Frederic Guintu, Rayvant Sahni · 2024

Read Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction by Hyunbae Jeon, Frederic Guintu, Rayvant Sahni (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18061.txt.

Introduction

Hyunbae Jeon, Frederic Guintu, and Rayvant Sahni built Lla-VAP in 2024, an ensemble system that predicts when a speaker in conversation will hand over the turn. Knowing turn-yield points matters for natural spoken dialogue systems. Their approach joins two model families inside an LSTM ensemble.

One side is a large language model, which reads the words and their meaning. The other is a voice activity projection model, which times speech with fine precision. Tests ran on two corpora, one of natural conversation and one of coached preference talks. The authors weigh where current models do well and where they fail, and argue the combined frame should predict turn points more robustly in both scripted and free talk.

Catalog record

CREATORHyunbae Jeon, Frederic Guintu, Rayvant Sahni
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction Hyunbae Jeon, Frederic Guintu, Rayvant Sahni Abstract Turn-taking prediction is the task of anticipating when the speaker in a conversation will yield their turn to another speaker to begin speaking. This project expands on existing strategies for turn-taking prediction by employing a multi-modal ensemble approach that integrates large language models (LLMs) and voice activity projection (VAP) models. By combining the linguistic capabilities of LLMs with the temporal precision of VAP models, we aim to improve the accuracy and efficiency of identifying TRPs in both scripted and unscripted conversational scenarios. Our methods are evaluated on the In-Conversation Corpus (ICC) and Coached Conversational Preference Elicitation (CCPE) datasets, highlighting the strengths and limitations of current models while proposing a potentially more robust framework for enhanced prediction. 1 Introduction Turn-taking is a fundamental mechanism in human communication, ensuring orderly and coherent verbal exchanges. It involves participants alternating roles as speaker and listener, guided by verbal and non-verbal cues that signal when a speaker’s turn is nearing its end. These signals create opportunities, known as Transition Relevance Places (TRPs), where the conversational floor can shift to another participant. TRPs are not fixed but dynamically emerge from the interplay of syntactic, semantic, prosodic, and contextual cues within a conversation Umair et al. 2024 . The ability to predict TRPs is important in the process of designing effective spoken dialogue systems (SDS) and conversational agents. Human interlocutors anticipate TRPs to avoid interruptions, minimize silence, and maintain conversational flow. However, artificial systems often struggle with this task, particularly in unscripted interactions where the timing and nature of TRPs vary significantly. Current models like TurnGPT and RC-TurnGPT demonstrate the potential of large language models (LLMs) for turn-taking tasks but remain limited to text-based features, lacking integration of acoustic cues critical for real-world conversations Ekstedt and Skantze 2020 . Our project addresses these limitations by proposing a multi-modal approach that leverages both linguistic and acoustic features. Specifically, we integrate a Large Language Model (LLM) with a Voice Activity Projection (VAP) model to capitalize on their complementary strengths. LLMs excel in understanding the syntactic and semantic context of conversations, while VAP models provide insights into non-verbal cues, such as pauses and tonal shifts, that signal TRPs. This combination aims to enhance the accuracy and robustness of turn-taking prediction across diverse conversational settings Wang et al. 2024 . Building on prior research, we utilize the In-Conversation Corpus (ICC) as a primary dataset to explore unscripted turn-taking dynamics. The ICC, with its participant-labeled within-turn TRPs, provides a nuanced understanding of conversational behavior and highlights the challenges LLMs face in predicting these nuanced interaction points. Unlike scripted datasets, the ICC reveals the variability and subtlety of natural conversations, making it an ideal testbed for our proposed methodology. Complementing this, we employ the Coached Conversational Preference Elicitation (CCPE) dataset to analyze turn-taking in controlled task-oriented dialogues, facilitating a comprehensive evaluation of our models Radlinski et al. 2019 . By bridging the gap between single-mode and multi-modal approaches, our research seeks to advance the state of turn-taking prediction. Beyond its technical implications, accurate TRP prediction holds broader significance for improving human-computer interaction, enabling conversational agents to engage in dialogues that feel more natural, fluid, and human-like. 2 Related Work With advancements in natural language processing came large language models (LLMs) which are computational systems that have demonstrated the ability to understand human language and its contextual nuances. These models can capture linguistic patterns that indicate conversational flow, as demonstrated by systems like TurnGPT, which is based on a transformer architecture fine-tuned for turn-taking prediction. TurnGPT effectively identifies turn-shift points by leveraging syntactic and pragmatic completeness cues, making it a valuable model for turn-prediction tasks Ekstedt and Skantze 2020 . However, the TurnGPT model’s understanding remains limited to text-based inputs, thus lacking integration of nonverbal cues critical for natural conversation dynamics. The Voice Activity Projection (VAP) model is a continuous turn-taking prediction system designed to anticipate future voice activity in conversations, contrasting traditional models that rely on simple silence thresholds. The VAP model uses multi-layer transformers to process raw audio signals and predict the likelihood of voice activity for each speaker over a specified time frame. This approach allows the model to detect subtle cues in conversational dynamics, such as pauses and shifts, essential for turn-taking decisions Inoue et al. 2024 . Recently, Wang et al. Wang et al. 2024 proposed an method to turn-taking prediction by combining a neural acoustic model with a large language model. Their experiments demonstrated that this multi-modal fusion of lexical and acoustic information "consistently outperforms baseline models with single modality," highlighting that an effective integration of both linguistic and audio cues enhances predictive accuracy. These findings indicate the potential of multi-modal approaches to significantly improve outcomes in turn-taking prediction when properly implemented Wang et al. 2024 . Table 1: Properties of CCPE and ICC dataset Aspect CCPE ICC Data Composition 502 English dialogues with 12,000 annotated utterances Approximately 93 conversations, each lasting about 25 minutes Format JSON format High-quality audio recordings Language English American English Source Google LLC Human Interaction Lab (HiLab) at Tufts University Purpose Elicit conversational preferences about movies in a controlled setting, minimizing bias in user language Study informal, unscripted dialogues to analyze turn-taking and identify multiple Turn Construction Units (TCUs) 3 Dataset For this project, we use two primary datasets: the Coached Conversational Preference Elicitation (CCPE) dataset Radlinski et al. 2019 and the In-Conversation Corpus (ICC) Umair et al. 2024 . 3.1 CCPE The Coached Conversational Preference Elicitation is designed specifically for eliciting conversational preferences in a controlled setting. It consists of 502 English dialogues with 12,000 annotated utterances between a user and an assistant discussing movie preferences in natural language. The corpus was constructed from dialogues between two paid crowd-workers using a Wizard-of-Oz methodology. One worker plays the role of an "assistant", while the other plays the role of a "user". The "assistant" is tasked with eliciting the "user" preferences about movies following a Coached Conversational Preference Elicitation (CCPE) methodology. In particular, the assistant is required to ask questions designed so as to minimize the bias in the terminology the "user" employs to convey his or her preferences, and obtain these in as natural language as possible. Each dialog is annotated with entity mentions, preferences expressed about entities, descriptions of entities provided, and other statements of entities. The dialogue corpus is provided in JSON format in a file called data.json. 3.2 ICC The In-Conversation Corpus consists of high-quality recordings of informal dialogues in American English, collected at the Human Interaction Lab (HiLab) at Tufts University. It features pairs of undergraduate students engaged in unscripted conversations, collected in a controlled environment with sound isolation to ensure clarity. The dataset was filtered to create a subset of turns suspected to contain multiple Turn Construction Units (TCUs), providing a rich source for identifying both turn-final and within-turn TRPs. In total, the ICC consists of approximately 93 conversations, each lasting approximately 25 minutes. These datasets will be employed to train and evaluate our models, providing diverse conversational scenarios to assess the effectiveness of our turn-taking prediction methods. 4 Approach Our study implements a multi-modal approach to turn-taking prediction that leverages

Read the complete text