Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment
Andrew Chang 1 , Viswadruth Akkaraju 1 , Ray McFadden Cogliano 1 , David Poeppel 1 2 , Dustin Freeman 1 · 2025
Read Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment by Andrew Chang 1 , Viswadruth Akkaraju 1 , Ray McFadden Cogliano 1 , David Poeppel 1 2 , Dustin Freeman 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03190.txt.
Introduction
Andrew Chang, David Poeppel, Dustin Freeman, and co-workers taught a model to predict when video calls feel awkward in a 2025 paper. Videoconferencing is routine, yet calls often lack the ease of face-to-face talk, and the bad moments are rare and hard to catch. The team sampled thousands of short clips from the RoomReader corpus of recorded calls, extracting audio embeddings, facial action. Body motion traits, then trained models to flag low conversational fluidity, low enjoyment, and set events like backchanneling, interruptions, and gaps.
The best models scored strongly on held-out calls, and general audio traits mattered most. Beyond videoconference design, the work shows multimodal signals can predict high-level subjective outcomes. Pointing to tools that catch a call going sour in time to fix it.
Catalog record
| CREATOR | Andrew Chang 1 , Viswadruth Akkaraju 1 , Ray McFadden Cogliano 1 , David Poeppel 1 2 , Dustin Freeman 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03190
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Machine Learning Can Predict Videoconference Fluidity and Enjoyment Thanks: A.C., D.P., and D.F. are supported by NYU Discovery Research Fund for Human Health. A.C. is supported by National Research Service Award, NIDCD/NIH (F32DC018205) and Leon Levy Scholarships in Neuroscience, Leon Levy Foundation and New York Academy of Sciences. The funders have no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise. © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Andrew Chang 1 , Viswadruth Akkaraju 1 , Ray McFadden Cogliano 1 , David Poeppel 1 2 , Dustin Freeman 1 Abstract Videoconferencing is now a frequent mode of communication in both professional and informal settings, yet it often lacks the fluidity and enjoyment of in-person conversation. This study leverages multimodal machine learning to predict moments of negative experience in videoconferencing. We sampled thousands of short clips from the RoomReader corpus [ 1 ] , extracting audio embeddings, facial actions, and body motion features to train models for identifying low conversational fluidity, low enjoyment, and classifying conversational events (backchanneling, interruption, or gap). Our best models achieved an ROC-AUC of up to 0.87 on hold-out videoconference sessions, with domain-general audio features proving most critical. This work demonstrates that multimodal audio-video signals can effectively predict high-level subjective conversational outcomes. In addition, this is a contribution to research on videoconferencing user experience by showing that multimodal machine learning can be used to identify rare moments of negative user experience for further study or mitigation. I Introduction and Goals Videoconferencing, as a medium, attempts to simulate in-person conversation. However, it often results in conversations that are less effective for fostering genuine interaction and intimacy, and “Zoom fatigue” is one of the well-known negative experience outcomes [ 2 , 3 ] . The causes of negative experience remain unclear and may involve factors such as signal delay/latency, reduction in signal quality due to compression, packet loss, and mismatch of perceptual cues. The bulk of time and data in any videoconferencing session is unproblematic and thus uninteresting for failure analysis. It is more useful to find specific moments of negative experience. Even if these events are infrequent, their negative impact on the conversation’s participants is disproportionate [ 4 ] . However, the definition of negative psychological experience is a high-level subjective outcome that usually requires manual, time-consuming annotation. Fig. 1: The flow diagram of the multimodal machine learning We aimed to determine how well a machine learning model trained on multimodal signals (audio, facial actions, body motion) can predict subjective conversational outcomes (i.e., fluidity and enjoyment) or classify conversational events (backchanneling, interruption, gap) in a short videoconference period (Fig. 1 ). Specifically, we examined: (1) the contribution of each feature domain to model performance, (2) whether the pre-event features can also predict outcomes, (3) the relationship between low-level fluidity and high-level enjoyment, and (4) the model’s ability to differentiate conversational events. To ensure generalizability, the trained model was tested on new videoconference sessions. II Prior Work Most prior research on applying machine learning to videoconferencing has pursued goals different from ours, focusing primarily on extending video analytics techniques to dynamic, real-world multiparty videoconferencing scenarios [ 5 ] , or facilitating signal quality [ 6 ] . Some studies were similar to those of the current work: one study used multimodal machine learning, trained on facial expressions, audio, eye gaze and more, predicted collaborative problem solving performance via videoconferencing [ 7 , 8 ] . Another study with a similar design showed that the combined video, speech and body motion features are statistically associated with subjective perception of the collaboration [ 9 ] . A recent study showed that nonverbal features can predict the next speaker in multiparty videoconference [ 10 ] . Another recent study experimentally manipulated different videoconference system parameters (signal delay, jitter, and packet loss rate), and demonstrated that a machine model trained on facial and speech features can predict participants’ quality of experience [ 11 ] . Our study distinguishes itself from previous research in several key ways: (1) it leverages a significantly larger dataset ( \sim 3000 vs. dozens to hundreds of sessions), (2) it models videoconferencing experiences within a much shorter timescale (7 s vs. \sim 15–60 mins), and (3) it focuses on high-level psychological and emotional dimensions of social interaction. III Dataset III-A Video Clips III-A 1 Videoconference Corpus: RoomReader We extracted the videoconference clips from the RoomReader corpus [ 1 ] . This corpus contains 30 Zoom videoconference sessions, each has 4 or 5 participants and last 8-30 mins, totalling near 9 hours and 118 participants (91 English native speakers). The conversations were mostly on collaborative quiz games and icebreakers. The procedure of the study was carefully designed to elicit naturalistic, spontaneous speech, i.e. turn-taking among participants based on their own initiative, without an explicit turn order. III-A 2 Clip Selection We aimed to reduce the clip count to a manageable size for rating and increase the representation of negative moments in model training. For this work, we chose to focus on changes in turn-taking, where either nobody was speaking or two or more people were speaking simultaneously. We implemented this selection according to the following rules: We set root-mean-square 0.05 as a binary threshold for detecting each speaker’s audio. When all speaker’s audio was below threshold for 0.75 s (much longer than the typical turn-taking transitional gap of 0.2 s in-person or 0.5 s on videoconference [ 12 ] ), we marked that as the beginning of a silence period. When the number of people speaking simultaneously exceeded one, we marked that as the beginning of an overlap period. A total of 1,508 marked time point in each category (3,016 clips in total) were sampled in the current study. For each marked time point, we extracted a group Zoom video clip from 3 s before to 4 s after. We did not highlight any information about how it was marked, or which speakers were involved to the raters. To help raters identify the moment of interest, we added a red border around each video clip from the marked time until 0.5 s after. III-B Clip Rating Survey III-B 1 Raters The raters were students at New York University who signed up for the studies via an online platform and received course credit for completing the experiments. In total, 528 raters (274 women, 167 men, 87 non-reporting or other) completed the survey, with ages ranging from 18 to 36 (median 20), and 242 (46%) identified as native English speakers. They reported a median of 4 hours of weekly video chat use, with time divided between friends and family (60%), study or work (12%), and classes or conferences (18%), showing a balance between informal and formal settings. III-B 2 Survey Procedure Participants completed the