SpeechCap: Leveraging Playful Impact Captions to Facilitate Interpersonal Communication in Social Virtual Reality
Yu Zhang, Yi Wen, Siying HU, Zhicong Lu · 2025
Read SpeechCap: Leveraging Playful Impact Captions to Facilitate Interpersonal Communication in Social Virtual Reality by Yu Zhang, Yi Wen, Siying HU, Zhicong Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10736.txt.
Introduction
Yu Zhang, Yi Wen, Siying Hu, and Zhicong Lu present SpeechCap, a playful caption system for chat in social virtual reality in two thousand twenty-five. Talk in shared virtual spaces still feels thin next to face-to-face chat. Impact captions, the bold text bursts used in videos, can carry tone and gesture as well as words.
The team first mapped how such captions can vary, with help from four experts in a co-design session. Their working system then turns speech into live impact captions inside VR. Fourteen users tried it in a study. The design let speakers blend verbal and non-verbal cues with ease, and the authors draw lessons on rhetoric, play, and useful ambiguity.
Catalog record
| CREATOR | Yu Zhang, Yi Wen, Siying HU, Zhicong Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpeechCap: Leveraging Playful Impact Captions to Facilitate Interpersonal Communication in Social Virtual Reality Yu Zhang, Yi Wen, Siying HU, Zhicong Lu Abstract . Social Virtual Reality (VR) emerges as a promising platform which affords immersive, interactive, and engaging mechanisms for collaborative activities in virtual spaces. However, interpersonal communication in social VR is still limited with existing mediums and channels. To bridge the gap, we propose a novel method for mediating real-time conversations in social VR, which leverages impact captions , a type of typographic visual effect widely used in videos, to encode both verbal and non-verbal information. We first investigated the design space of impact captions by content analysis and a co-design session with four experts. We then implemented SpeechCap, a proof-of-concept system with which users can communicate with each other using speech-driven impact captions in VR. Through a user study (N=14), we evaluated the effectiveness of the visual and interaction design of speech-driven impact captions, highlighting their strengths in the interactivity and integrating verbal and non-verbal information in communication mediums. Finally, we discussed our main findings regarding visual rhetoric, interactivity, and ambiguity, and further provided design implications for facilitating interpersonal communication in social VR. ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Virtual reality CCS: Human-centered computing Interactive systems and tools CCS: Human-centered computing Empirical studies in interaction design Figure 1. SpeechCap showcases a novel approach to mediate voice conversations in social VR by converting speech into playful impact captions. (A) From real-time speech, SpeechCap generates impact captions with customized visual design that conveys both verbal and non-verbal information. The caption “happy” is in a warm and bright color for the positive emotions, and a “smiling” emoji further highlight the pleasant and relaxing atmosphere. (B) Users can play with impact captions like real-world objects, e.g., throwing a “cake”. (C) Once the flying “cake” crashes on another user, it will explode like a firework. . 1. Introduction With the increasing availability of consumer-level virtual reality head-mounted displays (HMDs) ( Anthes et al., 2016 ; Li et al., 2021 ) , Social Virtual Reality (VR) emerges as a promising platform that brings immersive, interactive, and engaging experiences to users for a wide range of collaborative activities. Social VR now supports remote conferencing ( Abdullah et al., 2021 ) , immersive learning ( Thanyadit et al., 2022 ; Peng et al., 2022 ) , healthcare ( Li et al., 2020 ; Udapola, 2022 ) , collaborative design ( Mei et al., 2021 ) , music composing ( Men and Bryan-Kinns, 2022 ) and even daily activities such as sleeping ( Maloney and Freeman, 2020 ) and drinking ( Chen et al., 2024 ) . Among these diverse application areas, social VR helps users overcome physical dispersion, enabling them to communicate and collaborate in the same virtual environment ( Li et al., 2021 ; Palmer, 1995 ) by offering channels for both verbal and non-verbal communication ( Dzardanova et al., 2022 ; McVeigh-Schultz and Isbister, 2021 ; Wei et al., 2024 ) . Specifically, text messaging and voice chat are two channels commonly used for verbal communication. And pictures ( Li et al., 2019 ) , avatars ( Fu et al., 2023 ; Baker et al., 2021 ; Kolesnichenko et al., 2019 ; Freeman and Maloney, 2021 ) , and embodied interactions ( Maloney et al., 2020 ; Wieland et al., 2022 ) are channels that can convey non-verbal information in social VR. Yet, these existing communication channels in social VR are preliminary, as they are either borrowed from traditional media (e.g., texts, voices, and pictures) lacking adaptations to the nature of social VR ( Montoya Esquer and Lara López, 2023 ; Rzayev et al., 2021 ; Dzardanova et al., 2022 ; Hsieh et al., 2020 ) , or difficult to be reproduced very well by current VR technologies (e.g., gestures, facial expressions, embodied actions) ( Sykownik et al., 2023 ; Tanenbaum et al., 2020 ; Wei et al., 2024 ) . Specifically, texts as a communication medium in VR still suffer from difficulties in user inputting ( Montoya Esquer and Lara López, 2023 ) and readability issues in presentations ( Rzayev et al., 2021 ; Hsieh et al., 2020 ) . Voice-based conversations in VR can easily be disturbed by sound intrusions from the environment ( Akselrad et al., 2023 ) . Furthermore, non-verbal communication mediums, such as gestures and facial expressions, cannot be well-simulated with the avatars in current VR applications ( Baker et al., 2021 ; Aseeri and Interrante, 2021 ; Freeman and Maloney, 2021 ) , and users may find it hard to achieve specific actions or movements using controllers with joysticks and buttons ( Sykownik et al., 2023 ; Tanenbaum et al., 2020 ; Li et al., 2019 ) . Except for social VR, previous research in HCI and CSCW has explored ways to enhance interpersonal communication in different scenarios. Those scenarios include video conferencing ( Liu et al., 2023 ; Xia et al., 2023 ) , augmented reality (AR) ( Lee et al., 2023 ; Liao et al., 2022 ; Leong et al., 2022 ) , and text messaging interfaces ( Aoki et al., 2022 ) . These different scenarios demonstrate the power of the integration of both verbal and non-verbal cues in the mediums to support interpersonal communication. On the other hand, researchers have also envisioned the potential of social VR in providing situational awareness and enabling new social interactions with novel communication mediums that are unattainable in traditional computer-mediated communication (CMC) platforms ( McVeigh-Schultz and Isbister, 2021 ; McVeigh-Schultz and Isbister, 2022 ) . On top of the prior work, we proposed a new approach to support interpersonal communication in social VR. Our approach is inspired by Impact Captions , a type of expressive typographic 2D visual effect with engaging visual elements and animated motions. Impact captions are traditionally used in entertaining television shows and online videos ( O’Hagan, 2010 ; Sasamoto, 2014 ) to attract viewers’ attention and create engaging watching experiences ( Sasamoto et al., 2021 ) . Particularly, we introduced impact captions as a communication medium to facilitate interpersonal communication and interactions in social VR. In our design, impact captions encode both verbal and non-verbal information extracted from real-time speech; they also offer interactivity that allows users to intentionally play with the captions for communicative purposes. To implement and evaluate our impact-caption-inspired approach, we first explored the visual and interaction design space of impact captions through a content analysis on TV show videos. Based on the design space, we held a co-design session with 4 experts (i.e., HCI researchers) to further derive specific design goals for building a system to demonstrate our idea. Following the findings so far, we developed SpeechCap, a proof-of-concept system that takes real-time speech as input to generate interactive impact captions with customized visual appearances in a multi-user VR environment. Using SpeechCap, we conducted an in-lab study with 14 participants to assess impact captions as a medium for supporting interpersonal communication in social VR. The user study results highlighted the integration of verbal and non-verbal information offered by impact captions, indicating the design can make social VR communication experiences not only clear but also engaging, while the risks of miscommunication caused