Concert Interaction Translation: Augmenting VR Live Concert Experience using Chat-Driven Artificial Collective Reactions
Sebin Lee, Yeonho Cho, Jungjin Lee · 2025
Read Concert Interaction Translation: Augmenting VR Live Concert Experience using Chat-Driven Artificial Collective Reactions by Sebin Lee, Yeonho Cho, Jungjin Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13121.txt.
Introduction
Sebin Lee and co-authors bring the roar of a stadium crowd into sparse virtual reality concerts. Few fans own VR headsets, so virtual venues feel empty next to the shared thrill of a live show. Meanwhile, thousands chat along on streaming platforms, and that chat holds a live read of the crowd's mood.
Their system mines the chat for excitement and feeling, then turns the readings into motion and sound. Virtual audience avatars sway together, cheer, and sing along in step with the real crowd's energy. In a study with forty-eight users, the full pairing of movement and audio lifted immersion and the sense of being there together. Audio or motion alone did less, the team found.
Catalog record
| CREATOR | Sebin Lee, Yeonho Cho, Jungjin Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13121
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concert Interaction Translation: Augmenting VR Live Concert Experience using Chat-Driven Artificial Collective Reactions Sebin Lee, Yeonho Cho, Jungjin Lee Abstract . Computer-mediated concerts can be enjoyed on various devices, from desktop and mobile to VR devices, often supporting multiple devices simultaneously. However, due to the limited accessibility of VR devices, relatively small audience members tend to congregate in VR venues, resulting in diminished unique social experiences. To address this gap and enrich VR concert experiences, we present a novel approach that leverages non-VR user interaction data, specifically chat from audiences watching the same content on a live-streaming platform. Based on an analysis of audience reactions in offline concerts, we designed and prototyped a concert interaction translation system that extracts the level of engagement and emotions from chats and translates them to collective movements, cheers, and singalongs of virtual audience avatars in a VR venue. Our user study (n=48) demonstrates that our system, which combines both movement and audio reactions, significantly enhances the sense of immersion and co-presence than the previous method. Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706599.3720293 ISBN: 979-8-4007-1395-8/2025/04 CCS: Human-centered computing Interaction techniques CCS: Human-centered computing Virtual reality CCS: Human-centered computing User studies Figure 1. Concept of Concert Interaction Translation (CIT) system: The CIT system analyzes chat interactions produced by live-streaming audience members in real time and translates them into the collective reactions of dummy audience avatars in a VR concert venue to enhance the audience experience in VR. Conceptual Visualization of the Concert Interaction Translation System. Two side-by-side images compare a live-streamed concert performance (left) with chat messages on-screen, and a virtual reality concert setting (right) featuring audience avatars. The figure highlights how real-time chat interactions are translated into avatar reactions—movement, cheering, and singalong—to enhance immersion in the VR venue. 1. Introduction Live online concerts have emerged as a crucial platform for fandom businesses, particularly following the COVID-19 pandemic. They have also become an essential medium for newer entertainers, such as virtual YouTubers, to showcase their talents ( Lee and Lee, 2023 ; Morrissy, 2020 ; Amos, 2022 ) . These concerts are delivered through various methods, including livestreaming on platforms like YouTube and Twitch, where audiences engage via chat ( Vandenberg, 2022 ; Onderdijk et al., 2021 ) , as well as through VR venues like VRChat, which offer immersive experiences using head-mounted displays (HMDs) ( Lee et al., 2023a ; Lee and Lee, 2023 ) . Recently, some performers have adopted multiple delivery methods simultaneously to cater to diverse audience preferences ( Lee et al., 2023b ; Lin et al., 2023 ; Lee and Lee, 2023 ) . However, the delivery methods reveal a disparity in audience participation. Livestreams are widely accessible and tend to attract large audiences, while VR concerts, although highly immersive, appeal to a niche group due to the requirement of HMDs ( Lee and Lee, 2023 ) . Furthermore, VR platforms often limit the number of users per session to ensure technical stability. These limitations result in smaller audience sizes in VR, which diminishes the potential to cultivate the social experiences crucial for enhancing audience engagement in the entertainment domain ( Otsu et al., 2021 ; Mulder and Hitters, 2023 ) . Researchers have explored various computational methods to address the lack of audiences in VR musical performances. One common approach involves augmenting visual elements, such as adding non-player character (NPC) audiences or triggering visual effects based on biosignals that estimate emotional states ( Park et al., 2024 ; Meng et al., 2023 ; Muñoz-González et al., 2022 ; Ide and Horie, 2022 ; Horie et al., 2018 ; Lee et al., 2022 ) . While these methods provide emotionally accurate reflections, they require additional sensing hardware, limiting their practicality in real-world applications. Other approaches have focused on leveraging contextual information within the concert itself. For example, Yakura and Goto proposed synthesizing cheering motions of dummy avatars based on the acoustic features of concert music ( Yakura and Goto, 2020 ) . However, the emotions conveyed by music do not always align with the audience’s actual feelings ( Kawakami et al., 2012 ; Evans and Schubert, 2008 ) . This misalignment can result in artificial audience reactions that differ from the true emotional states of the participants. Furthermore, previous studies have primarily focused on leveraging visual elements, overlooking auditory responses like cheers and singalongs, which are integral to offline concert experiences. In this paper, inspired by the observation that live online concerts often utilize multiple delivery methods simultaneously ( Lee et al., 2023b ; Lin et al., 2023 ; Lee and Lee, 2023 ) , we introduce the Concert Interaction Translation (CIT) system that uses audience reactions from live-streaming concerts to enhance VR live concert experiences (Figure 1 ). The system translates chat interactions into the collective reactions of dummy audience avatars in VR, replicating both verbal and non-verbal audience responses commonly seen in offline concerts. To achieve this, we first analyzed 68 concert videos to identify representative movement and sound reactions based on the emotional arousal and valence of the audience. Subsequently, we designed the CIT system that estimates engagement and emotional level from live-streaming concert chats in real time and synthesizes movement, cheers, and singalongs in VR. A user study using a proof-of-concept prototype of CIT demonstrated that the CIT significantly enhances immersion and co-presence compared to the previous method that relied solely on acoustic features of concert music ( Yakura and Goto, 2020 ) . Table 1. Description of classified movement and sound reactions according to the arousal and valence level Description of Movement and Sound Reactions by Arousal and Valence. This table categorizes audience reactions into three arousal levels (high, neutral, low) and three valence levels (negative, neutral, positive). It outlines specific arm movements—such as shaking arms back and forth or side to side—and vocal expressions—like cheering or disappointed sounds—based on each combination of arousal and valence, with references to relevant figures and external timestamps. High Arousal Movement: Disappointing Motion (Figure 2(d) ) Sound: Disappointing Sound: Ahhhh… (cf. ( U.F.O, 2019 ) (0:42–0:50)) Movement: In fast tempo: Shaking Arms: Back and Forth (Figure 2(a) ) In low tempo: Shaking Arms: Side by Side (Figure 2(b) ) When talking toward audience: Hand clap (Figure 2(c) ) Sound: Shouting: “Waaaaa,” “Woooow,” or “Hoooou” (cf. ( Coldplay, 2018 ; officialpsy, 2012 ) (0:00–0:10)) Neutral Arousal Movement: Hand clap (Figure 2(c) ) Sound: Hand clap (cf. ( MAGIC STRAWBERRY SOUND, 2018 ) (0:30–0:40)) Low Arousal Not Assigned Low Valence (Negative Emotion) Neutral Valence High Valence (Positive Emotion) 2. Analysis of Audience Reactions in Offline Concerts We first analyzed offline concert footage to understand audience reactions in the wild. We collected footage from YouTube using keywords such as “Live Concert,” “Concert,” and “Concert Fancam.” Additionally, we included a complete concert video from a Blu-ray disc to examine reactions during singing, performances, and conversations. We gathered 68 videos spanning 12 hours and 25 minutes and extracted 3 hours and 9 minutes of audience-focused clips. These clips were qualitatively