Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detection
Myeonghoon Ryu ∗†1 , June-Woo Kim ∗†2,3 , Minseok Oh 1 , Suji Lee 1 , Han Park 1 · 2025
Read Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detection by Myeonghoon Ryu ∗†1 , June-Woo Kim ∗†2,3 , Minseok Oh 1 , Suji Lee 1 , Han Park 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11631.txt.
Introduction
Myeonghoon Ryu and co-authors harden a cry-for-help detector against the noise of real homes. Listening systems that spot calls for help save lives, but microphones and room noise trigger false alarms once deployed. Retraining for every new phrase is also a burden.
Their approach starts from a ready-made speech recognizer, Whisper, and adds a second job: judging the noise itself. Training the model on both tasks, in a way that refuses to lean on noise identity, steadies its ear. The team reports a sharp drop in false alarms and better overall detection, at modest computing cost. The design keeps the door open to new keywords without rebuilding the system.
Catalog record
| CREATOR | Myeonghoon Ryu ∗†1 , June-Woo Kim ∗†2,3 , Minseok Oh 1 , Suji Lee 1 , Han Park 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detection Myeonghoon Ryu ∗†1 , June-Woo Kim ∗†2,3 , Minseok Oh 1 , Suji Lee 1 , Han Park 1 Abstract Keyword spotting is often implemented by keyword classifier to the encoder in acoustic models, enabling the classification of predefined or open vocabulary keywords. Although keyword spotting is a crucial task in various applications and can be extended to call-for-help detection in emergencies, however, the previous method often suffers from scalability limitations due to retraining required to introduce new keywords or adapt to changing contexts. We explore a simple yet effective approach that leverages off-the-shelf pretrained ASR models to address these challenges, especially in call-for-help detection scenarios. Furthermore, we observed a substantial increase in false alarms when deploying call-for-help detection system in real-world scenarios due to noise introduced by microphones or different environments. To address this, we propose a novel noise-agnostic multitask learning approach that integrates a noise classification head into the ASR encoder. Our method enhances the model’s robustness to noisy environments, leading to a significant reduction in false alarms and improved overall call-for-help performance. Despite the added complexity of multitask learning, our approach is computationally efficient and provides a promising solution for call-for-help detection in real-world scenarios. I Introduction ASR technology has emerged as a pivotal component in contemporary human-computer interaction, enabling applications ranging from virtual assistants to assistive solutions for individuals with hearing impairments. Among the various sub-tasks in ASR, keyword spotting (KWS) has gained considerable attention due to its critical role in enabling voice-activated systems [ 4 , 9 ] . While previous KWS studies has shown promise in various applications [ 4 , 27 , 19 , 33 , 23 ] , its effectiveness in detecting call-for-help in hazardous environments remains underexplored. We aim to address this gap by leveraging ASR and KWS concepts to develop an effective system for call-for-help detection in real-world scenarios [ 6 ] . KWS specifically involves detecting predefined words or phrases within a continuous speech stream, serving as a trigger for executing predetermined actions, such as activating smart speakers or initiating command sequences [ 31 ] . Typically, a keyword classification head is attached to the encoder of an acoustic model to identify predefined keywords. While this method has demonstrated efficacy in finite keyword settings, such as query-by-example [ 5 , 28 , 14 , 11 , 32 ] , it frequently encounters challenges when deployed in real-world environments. One key limitation is scalability. Previous approaches require retraining when new keywords are introduced or when the system needs to adapt to different contexts. This becomes particularly problematic in dynamic environments where new keywords may emerge. Consequently, this constant retraining can lead to inefficiencies for KWS service. To address these issues, studies on Open-Vocabulary KWS system were emerged [ 21 , 19 , 3 , 33 ] , including few-shot trials [ 20 ] , leveraging speech embeddings [ 17 ] and text-audio embeddings [ 27 , 29 , 23 ] . Despite advancements in scalability, their works often involve complex network architectures and rely on cascaded ASR systems for KWS, which can introduce challenges for user implementation. This paper proposes a simple yet effective approach for KWS, particularly focused on detecting calls-for-help in emergency situations. By fine-tuning an off-the-shelf large-scale Whisper [ 25 ] , on a small dataset specific to emergency-related calls for help, we aim to develop a more adaptive and scalable KWS system. Our method leverages ASR outputs to enable quick adaptation to new keywords without the need for retraining, making it highly suitable for real-world deployment. However, deploying systems in real-world environments is a challenging tasks, especially due to noise [ 12 , 5 , 34 , 11 , 22 ] . Microphones in devices like smartphones and smart speakers often pick up substantial background noise, which can significantly impair system performance. For the call-for-help in emergency situations, where both accuracy and speed of ASR are critical, false alarms triggered by noisy conditions can have serious consequences. We introduce a multitask learning approach that integrates a noise classification head into the Whisper encoder. This multitask setup enables the model to learn noise patterns that commonly trigger false alarms, enhancing its robustness in noisy environments while using only a small number of additional model parameters. By simultaneously training the model to recognize both keywords and environmental noise types, our model can be a noise-agnostic call-for-help detection system in practical scenarios. Experimental results validate that our proposed method improves the model’s robustness in noisy environments, significantly reducing false alarms and enhancing overall performance in real-world scenarios. Our contributions are summarized below: • We present a simple yet effective approach to call-for-help detection in emergency situations. By leveraging off-the-shelf Whisper for fine-tuning, we address the scalability issues present in previous KWS systems. • We propose a multitask learning approach for a noise-agnostic call-for-help detection system that requires only a few additional model parameters. Our method significantly reduces false alarms while enhancing overall performance in noisy environments, offering a practical solution for emergency response. • To validate the effectiveness of our method in real-world scenarios, we recruited participants and recorded call-for-help situations. As a result, our method significantly reduced false alarms in diverse environments. Finally, we provide the code to the KWS community to support further research and development. II Method II-A Fine-tuning Whisper for call-for-help detection Whisper [ 25 ] is a large-scale pretrained ASR model trained on 680K hours of multilingual speech data. Besides, Whisper is further trained on various speech processing tasks using a multitask supervision approach, such as translation, voice activity detection, and language identification, etc. By leveraging the pretrained Whisper model, it can be easily fine-tuned to extend its rich knowledge for several downstream tasks. Fig. 1 : Overall illustration of proposed multitask learning approach that integrates a noise classification head into the Whisper encoder. We use the Whisper model for fine-tuning on call-for-help detection tasks, which are closely related to keyword spotting. Unlike previous approaches that rely heavily on predefined keywords or acoustic models, we categorize call-for-help scenarios and utilize ASR results for classification. In this work, we define emergency scenarios as: 1) shouting for help in life-threatening situations 2) crying out for assistance when personal safety is at risk and 3) other similar situations , though not limited to these. Based on the speech and its corresponding text, e.g., help me or save me , we classify these as emergency tasks. All other results are categorized as others . II-B Multitask training for reducing false alarm errors To alleviate the false alarm errors in noisy real-world scenarios, we introduce multitask learning for noise classification. As shown in Fig. 1 , noise classifier is added to the Whisper encoder, therefore the ASR encoder is sharable for noise classification as well as speech recognition. Therefore, the final loss \mathcal{L}_{\text{Multi}} can be formularized as: Fig. 2 : Overview of the proposed call-for-help detection system using our approach in real-world scenarios. \displaystyle\mathcal{L}_{\text{Multi}}=\mathcal{L}_{\text{Noise}}+\mathcal{L}_{\text{Seq2Seq}} (1) where the noise loss \mathcal{L}_{\text{Noise}} and ASR loss \mathcal{L}_{\text{Seq2Seq}} are: \displaystyle\mathcal{L}_{\text{Noise}}\!=-\sum_{i=1}^{n}\!\,m_{i}\!\,\log\,\!(\hat{m_{i}}), (2) \displaystyle\mathcal{L}_{\text{Seq2Seq}}\!=-\sum_{t=1}^{T}\!\,\log P(y_{t}|y_{1},...,y_{t-1},\mathbf{x}). (3) Here, both \mathcal{L}_{\text{Noise}} and \mathcal{L}_{\text{Seq2Seq}} are based on Cross-Entropy loss, with multitask noise label m and ASR