A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results
Thai-Binh Nguyen ⋆ , Katerina Zmolikova † , Pingchuan Ma † , Ngoc Quan Pham ‡ , Christian Fuegen † , Alexander Waibel ⋆‡ · 2025
Read A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results by Thai-Binh Nguyen ⋆ , Katerina Zmolikova † , Pingchuan Ma † , Ngoc Quan Pham ‡ , Christian Fuegen † , Alexander Waibel ⋆‡ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23276.txt.
Introduction
Thai-Binh Nguyen and five co-authors introduce a cocktail-party benchmark for the ninth CHiME Challenge. The task, Multi-Modal Context-Aware Recognition, asks who speaks when, what they say, and with whom, in a single room with overlapping casual conversation. Recordings show extreme overlap up to one hundred percent and highly broken turns.
Systems must fuse audio, visual, and contextual cues to transcribe each speaker and cluster their speech. Unscripted group chat makes the data far harder than clean studio speech. The benchmark pushes meeting transcription toward real social settings where everyone talks at once.
Catalog record
| CREATOR | Thai-Binh Nguyen ⋆ , Katerina Zmolikova † , Pingchuan Ma † , Ngoc Quan Pham ‡ , Christian Fuegen † , Alexander Waibel ⋆‡ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results Thai-Binh Nguyen ⋆ , Katerina Zmolikova † , Pingchuan Ma † , Ngoc Quan Pham ‡ , Christian Fuegen † , Alexander Waibel ⋆‡ Abstract We introduce the task of Multi-Modal Context-Aware Recognition (MCoRec) in the ninth CHiME Challenge, which addresses the cocktail-party problem of overlapping conversations in a single-room setting using audio, visual, and contextual cues. MCoRec captures natural multi-party conversations where the recordings focus on unscripted, casual group chats, leading to extreme speech overlap of up to 100% and highly fragmented conversational turns. The task requires systems to answer the question “Who speaks when, what, and with whom?” by jointly transcribing each speaker’s speech and clustering them into their respective conversations from audio-visual recordings. Audio-only baselines exceed 100% word error rate, whereas incorporating visual cues yields substantial 50% improvements, highlighting the importance of multi-modality. In this manuscript, we present the motivation behind the task, outline the data collection process, and report the baseline systems developed for the MCoRec. † † address: ⋆ Karlsruhe Institute of Technology, Germany † Meta AI, UK ‡ Carnegie Mellon University, USA [email protected] 1 Introduction Automatic speech recognition (ASR) has achieved significant improvements in noisy and reverberant conditions over the past few decades, supported by advances in deep learning, signal processing, and the availability of large real-world corpora [ 1 , 2 , 3 , 4 ] . However, ASR performance in natural multi-party conversations remains limited, particularly under overlapping speech and conversational dynamics, also known as the cocktail-party setting [ 5 , 6 , 7 , 8 , 9 ] . Although many approaches have been proposed to improve robustness through acoustic modeling, word error rates (WER) remain above 20% in such scenarios. In addition to research on acoustic robustness, audio-visual speech recognition (AVSR) has emerged as a promising approach by leveraging visual cues such as lip movements to complement the acoustic signal [ 10 , 11 , 12 , 13 , 14 ] . To advance this direction, several audio-visual and multi-modal datasets have been introduced. Large-scale corpora such as LRS2/LRS3 [ 15 , 16 ] , Muavic [ 17 ] focus primarily on single-speaker or Ego4D [ 18 ] broad egocentric scenarios, while conversational datasets such as AMI [ 9 ] , MISP [ 19 ] , and MMCSG [ 20 ] capture multi-party meetings but are limited to single conversations per session. CHiME-5/6 [ 5 ] capture multi-party meetings but remain limited to audio-only. Despite these efforts, there remains a clear gap, audio-visual, natural, unscripted, multi-party interaction under cocktail-party conditions. The challenge in such settings arises from the so-called cocktail-party phenomenon, which describes humans’ ability to selectively attend to one conversation while ignoring others. In this context, systems must determine not only who speaks when and what is spoken, but also with whom the interaction occurs. The MCoRec task aims to bridge the gap between previous efforts by establishing a benchmark for the cocktail-party challenge. Our dataset features audio and video recordings of up to eight participants engaged in up to four simultaneous conversations. The challenge provides a single 360° video and a single-channel audio recording, reflecting typical conditions using commercially available devices. Benchmarking systems will focus on jointly transcribing speech and clustering speakers into conversation groups. The MCoRec dataset and baseline systems have been made reproducible via https://github.com/MCoRec/mcorec_baseline 2 Dataset 2.1 Data collection The recordings were conducted in natural conversational settings where participants were seated in groups around a large table. Each group remained fixed throughout a session in terms of both participants and discussion topics to ensure consistency, while allowing spontaneous and natural interactions. Within a session, groups could either sit across from each other or in separate clusters, simulating realistic social gatherings. All participants were familiar with each other to encourage fluent and unconstrained conversations. Discussion topics covered a wide range of everyday themes, including personal life, hobbies, school, work, entertainment, news, and hypothetical situations. Data collection took place across 10 different indoor environments of varying size and type, including living rooms, meeting rooms, lecture halls, and other common spaces, thereby capturing diverse acoustic and visual conditions. Each session involved up to 8 speakers and up to 4 simultaneous conversations. Figure 1: Recording setup of a session in the MCoRec dataset. Figure 1 illustrates a recording scene with two groups, one in black and the other in white, seated across from each other. Each recording session was captured using a combination of central and participant-specific devices. A 360° camera (GoPro Max) was positioned at the center of the table at seated eye level, providing a panoramic view of all participants at 4K resolution. The integrated microphone of the 360° camera was used to record the overall session audio, with distances between the camera and participants ranging from approximately 50 cm to 2 m. Simultaneously, each participant was recorded with a personal setup consisting of a smartphone camera (720p resolution) connected to a lavalier microphone (AGPTEK Z02) placed near the mouth to ensure high-quality voice capture. These cameras were positioned below the central 360° camera, below mouth level, to avoid obstructing the central view while still capturing clear visual information of the speaker’s face and mouth region. A moderator signaled the beginning and end of each session using a whistle. Each recording session lasted approximately 6 minutes. 2.2 Data annotation The annotation process was carried out in 4 main steps. In the first step, we temporally aligned all recordings using the whistle signal that marked the beginning and end of each session. Annotators manually marked the approximate region containing the whistle, after which we refined the alignment automatically. Specifically, we computed the spectral-flux onset strength envelope with the librosa library (using the onset\_strength function) and identified the frame with the highest peak to pinpoint the exact whistle timestamp. This procedure ensured precise synchronization across all recording devices. The second step focuses on transcribing the speakers’ utterances. Annotators rely on the smartphone video and lapel microphone recordings, which provide the highest-quality audio, to ensure accurate transcription. Speech is manually segmented into utterances of no more than 15 seconds, each transcribed by the annotator. If a segment is unintelligible or cannot be reliably transcribed, it is marked with three asterisks (***) to indicate uncertainty. In the third step, we performed face tracking on the 360° video to spatially locate all target participants. Since raw 360° footage is stored in a dual-fisheye format, we stitched it into a single equirectangular projection using GoPro Player. The resulting video, exported as an MP4 file with stereo audio, was used as the input for the whole recognition system. For simplicity, we continue to refer to this stitched video as the 360° video. We followed the pipeline of [ 21 ] , which involves shot detection, face detection, and multi-frame tracking. This yielded a sequence of bounding boxes (top-left and bottom-right coordinates) that indicate the face of each participant throughout the recording. This step was necessary for two reasons. Firstly, not all individuals appearing in the 360° video were part of the session, so bounding boxes allowed us to restrict attention to actual participants. Secondly, face tracking enabled the correct mapping of transcriptions from Step 2