Leveraging Audio Representations for Vibration-Based Crowd Monitoring in Stadiums
Yen Cheng Chang, Jesse Codling, Yiwen Dong, Jiale Zhang, Jiasi Chen, Hae Young Noh, Pei Zhang · 2025
Read Leveraging Audio Representations for Vibration-Based Crowd Monitoring in Stadiums by Yen Cheng Chang, Jesse Codling, Yiwen Dong, Jiale Zhang, Jiasi Chen, Hae Young Noh, Pei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17646.txt.
Introduction
Yen Cheng Chang and co-authors presented a way to monitor crowds in stadiums from floor vibration in two thousand twenty-five. Cameras and microphones can disturb fans and raise privacy worries. Floor vibration is quieter, but labeled vibration data from huge stadiums are scarce.
Their method, ViLA, first learns from sound data with no labels, then adapts with a small amount of vibration data. Sound and vibration are both waves, so lessons from public audio sets carry over. In real-world tests, pre-training on a large public audio set cut error by up to about six times versus no pre-training. The result points to cheaper crowd sensing with less intrusion.
Catalog record
| CREATOR | Yen Cheng Chang, Jesse Codling, Yiwen Dong, Jiale Zhang, Jiasi Chen, Hae Young Noh, Pei Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Audio Representations for Vibration-Based Crowd Monitoring in Stadiums Yen Cheng Chang, Jesse Codling, Yiwen Dong, Jiale Zhang, Jiasi Chen, Hae Young Noh, Pei Zhang Abstract Crowd monitoring in sports stadiums is important to enhance public safety and improve audience experience. Existing approaches mainly rely on cameras and microphones, which can cause significant disturbances and often raise privacy concerns. In this paper, we sense floor vibration, which provides a less disruptive and more non-intrusive way of crowd sensing, to predict crowd behavior. However, since the vibration-based crowd monitoring approach is newly developed, one main challenge is the lack of training data due to sports stadiums are usually large public spaces with complex physical activities. In this paper, we present Vi bration L everage A udio (ViLA) , a vibration-based method that reduces the dependency on labeled data by pre-training with unlabeled cross-modality data. ViLA is first pretrained on audio data in an unsupervised manner and then fine-tuned with a minimal amount of in-domain vibration data. By leveraging publicly available audio datasets, ViLA learns the wave behaviors from audio and then adapts the representation to vibration, reducing the reliance on domain-specific vibration data. Our real-world experiments demonstrate that pre-training the vibration model using publicly available audio data (YouTube8M) achieved up to a 5.8 × \times error reduction compared to the model without audio pre-training. Figure 1: An overview of the football stadium. The complex crowd dynamics and high noise levels make crowd monitoring a challenging task. 1 Introduction Effective crowd monitoring enhances safety, security, and overall audience enjoyment at large-scale public events such as sports games [ 1 , 2 , 3 ] . Traditional methods like manual observation [ 4 ] and the deployment of various sensing systems [ 5 , 6 , 7 , 8 ] have been successful in monitoring audience activities. However, they face challenges such as requiring intensive manual labor, visual occlusions (computer vision), and insufficient accuracy due to time-varying noises from large groups (audio and WiFi/radio frequency) [ 9 , 10 , 11 ] . In recent years, floor vibration sensing has emerged as a cost-efficient, non-intrusive, and privacy-friendly alternative, facilitating continuous and fine-grained monitoring of crowd behaviors [ 12 , 13 , 14 ] . This new approach detects and analyzes crowed-induced vibrations without capturing an individual’s voice or appearance, offering detailed insights into crowd traffic and movement patterns while operating unobtrusively. However, collecting high-quality, well-labeled floor vibration data presents significant challenges due to the crowded environment at sports stadiums. First, deploying and synchronizing numerous sensors in large-scale, crowded spaces is time-consuming and complicated [ 15 , 16 , 17 ] . Accurately calibrating and positioning these sensors is essential but challenging due to heavy foot traffic and interference from environmental noise [ 18 , 19 , 20 , 21 , 22 ] . Additionally, floor vibrations are influenced by various factors such as material, weight, and location, which complicates the training of a general-purpose model that can cover all floor types and locations [ 23 , 24 , 18 ] . Figure 1 illustrates an example of large crowd monitoring. Thus, collecting data in such environments requires deploying numerous sensors, which face significant connectivity challenges in crowded settings [ 22 ] . To the best of our knowledge, there is a lack of well-labeled datasets to allow efficient training of vibration-based models, which mainly use self-collected, small-scale datasets and focus on a single person’s behaviors [ 18 , 25 , 11 , 26 ] . Besides, some popular supervised techniques like transfer learning and domain adaptation can utilize these data for crowd monitoring, but they still require substantial labeled data for effective pre-training [ 27 , 28 , 29 ] . Figure 2: Examples of audio and vibration spectrograms. The horizontal axis represents time (seconds), and the vertical axis represents frequency. Both spectrograms exhibit many similar patterns. In this study, we propose ViLA , a novel approach that leverages out-of-domain yet widely available datasets to pre-train the vibration-based crowd monitoring model. To compensate for the insufficient amount of vibration data, our approach leverages the abundant data resource from the chosen modality (i.e., audio) to capture the shared features in their time-frequency domain. Specifically, we first pre-train the crowd monitoring model using a publicly available audio dataset to capture basic features shared with the vibration data. Figure 2 shows some visual similarity between audio and vibration. Then, we fine-tune the model using a small amount of labeled vibration data for crowd monitoring. The main benefit of our approach is that we reduce the reliance on large-scale labeled training data, which significantly lessens the manual effort for sensor deployment, data collection, and labeling. The real-world evaluation at sports stadiums shows that using publicly available audio data (YouTube8M) for pre-training achieved up to a 5.8 \times reduction in error for vibration-based crowd behavior monitoring compared to the model without audio pre-training. This paper introduces a novel approach to pre-train a vibration-based crowd monitoring model using a publicly available audio dataset. The main contributions of this paper are: (1) Proposed Framework: We propose a ViLA , a framework that leverages out-of-domain modality (i.e., audio) to improve the performance of vibration-based models for crowd monitoring. (2) Definitions of the two indicators: We introduce two indicators, Similarity and Diversity , to evaluate the selection of modalities from various publicly available datasets to effectively enhance the performance of vibration-based models. (3) Real-World Evaluation: We demonstrate the effectiveness of our approach through real-world evaluations at two sports stadiums, which show promising results. We evaluate our approach by 1) analyzing various publicly available datasets, including audio, video, and image, and 2) conducting real-world experiments across two sports stadiums to collect crowd-induced floor vibration data. Our results show that audio signals enhance vibration-based models more effectively than other modalities using the Similarity and Diversity indicators. The real-world evaluation at sports stadiums shows that using publicly available audio data (YouTube8M) for pre-training achieved up to a 5.8 \times reduction in error for vibration-based crowd behavior monitoring compared to the model without audio pre-training. This demonstrates the effectiveness of our approach for efficient vibration-based model training and accurate crowd monitoring at sports stadiums. The rest of the paper is organized as follows: we demonstrate how we select audio as our pre-training modality (Section 2). Then, we illustrate the framework that leverages unlabeled, pre-existing audio datasets to enhance vibration-based models (Section 3). Subsequently, we conduct real-world experiments in two different environments (Section 4). Finally, we summarize the study (Section 5). 2 Unsupervised Pre-training Using Audio Data for Vibration Modeling In this section, we introduce a novel framework for vibration-based crowd monitoring model training that significantly reduces the reliance on labeled vibration data. Our approach consists of two steps: 1) unsupervised pre-training using public audio data and 2) supervised fine-tuning using minimal labeled vibration data specific to the target domain. This approach effectively addresses challenges such as data scarcity in harsh environments and reduces the effort and costs associated with data collection. By employing cross-domain learning techniques, our method not only bridges the gap between audio and vibration data but also enhances the efficiency and scalability of vibration monitoring systems. Figure 3: The overview of the ViLA