Lumename: Wearable Device for Hearing Impaired w/ Personalized ML-Based Auditory Detection and Haptic-Visual Alerts

Jeanelle Dao, Jadelynn Dao · 2025

Read Lumename: Wearable Device for Hearing Impaired w/ Personalized ML-Based Auditory Detection and Haptic-Visual Alerts by Jeanelle Dao, Jadelynn Dao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01576.txt.

Introduction

Jeanelle Dao and Jadelynn Dao present a smartwatch, in a 2025 preprint, that alerts deaf and hard-of-hearing wearers when their name is spoken. Hundreds of millions of people live with serious hearing loss. Catching a spoken name in a busy room is a daily problem for them. Their device, Lumename, listens on the watch itself and answers with vibration plus a visual signal. The model must be tiny and quick on cheap hardware.

Training data was the bottleneck, since one user's voice samples are few. The authors reshaped those samples with audio tricks to mimic other ages and voices. A search over model settings found a compact design with low power use. The finished watch ran on an Arduino-based board. Its name detection stayed above ninety percent accurate. The paper frames the device as a low-cost aid that works without a phone or cloud link.

Catalog record

CREATORJeanelle Dao, Jadelynn Dao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01576

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lumename: Wearable Device for Hearing Impaired w/ Personalized ML-Based Auditory Detection and Haptic-Visual Alerts Jeanelle Dao, Jadelynn Dao Abstract According to the World Health Organization, 430 million people experience disabling hearing loss. For them, recognizing spoken commands such as one’s name is difficult. To address this issue, Lumename, a real-time smartwatch, utilizes on-device machine learning to detect a user-customized name before generating a haptic-visual alert. During training, to overcome the need for large datasets, Lumename uses novel audio modulation techniques to augment samples from one user and generate additional samples to represent diverse genders and ages. Constrained random iterations were used to find optimal parameters within the model architecture. This approach resulted in a low-resource and low-power TinyML model that could quickly infer various keyword samples while remaining 91.67% accurate on a custom-built smartwatch based on an Arduino Nano 33 BLE Sense. 1 Introduction 1.1 Background and Significance 430 million people, including 34 million children, have disabling hearing loss [ 12 ] . This makes everyday communication and interaction particularly difficult. This is increasingly difficult when those with hearing loss cannot see who are talking to them, particularly when being called. The lack of visual cues worsens the communication barrier, further impacting everyday activities. 1.2 Objectives The objective of this project is to assist individuals with hearing loss in recognizing spoken commands by utilizing on-device machine learning to detect a user’s customized keyword via a real-time smartwatch. Subsequently, a haptic and visual alert will be issued to alert the user. The custom keyword will be chosen by the user, and the user will be prompted to upload recordings of their chosen keyword (ex. their specific name) to the cloud for training. Therefore, the framework to build the model must be general enough to work for any given word, yet the final model should be specific enough to avoid false positives. To be effective, this device must be at least 85% accurate when testing how well it identifies the correct keyword (true positives) and ignores unrelated sounds (true negatives). Computing resources must be consciously limited to ensure the device can run for a minimum of eight hours and respond to the keyword within three seconds without internet access. 2 Methods 2.1 Data Preprocessing Samples are 16-bit at 1kHz audio data that are organized into eight total subclasses. The two parent classes are “name” and “not name,” and each class contains four subclasses. “Name” is split based on the method of audio transformation: pitch shift , ambiance mixing , both , or neither . However, other transformations such as speed, volume, and timing (which shifts the audio to play earlier or later than the original) are present in every “name” sample. Conversely, “not name” is split into different types of background noise: static , ambiance , words , and words with ambiance mixing . The words subclass is made of random samples from the Google Speech Commands dataset [ 10 ] . Each of these word samples are transformed similarly to the “name” samples. The static samples are generated from the microphone used in the final device, and serve as a baseline recording of silence. Originally, the "name" dataset only consists of the limited samples recorded by the user. While audio augmentation is common practice to expand datasets [ 4 ] , it is not commonly used to represent completely different voices. However, for Lumename, an audio manipulation code transforms the user’s samples into a diverse dataset emulating additional genders and ages. The program randomly shifts a sample’s pitch, speed, volume, and timing (within a one second segment). Additionally, royalty-free ambiance sounds [ 7 ] , which are permitted for use under Pixabay’s Content License, are mixed into samples to train for different sets of background noise (see Figure 1 ). Figure 1: sample waveform shifted by audio manipulation code Additionally, samples are uploaded to the cloud into the training framework, Edge Impulse [ 3 ] . Using the Edge Impulse API, samples are then preprocessed using mel-frequency cepstral coefficients (MFCC), which transforms data into a representation that highlights important features and removes insignificant data [ 1 ] . Utilizing a constrained random .json file, twelve different combinations of MFCC coefficient parameters were tested, with a focus on frame length, frame stride, and number of filters. The final parameters (0.032s, 0.032s, and 40 filters respectively) contained the best balance between containing a sufficient amount of data, preventing overfitting, and reducing necessary compute power. 2.2 Model Training As the model architecture would be run directly on a limited-resource microcontroller, the model needed to be created with maximal simplicity while remaining accurate, which was mainly achieved by adjusting the number of layers used [ 9 ] . Therefore, by leveraging a constrained random .json file, over 72 combinations of low-resource layers were trained using Edge Impulse. After testing both 1D and 2D convolution, 1D convolution ultimately performed well and required less computing resources. Within both convolution types, different amounts of filters were also tested. Dropout layers were also utilized to prevent overfitting, with the dropout rate being adjusted within the constrained random loop. To determine which sets of parameters performed better, a validation set was constructed using non-user generated samples. This ensured that the final model was general enough to detect when anyone said the keyword, not just the user. Additional static, ambiance, and Google Speech Command samples (that were not in the training set) were also included in the aforementioned validation set. To determine performance, a standard confusion matrix is generated using the subclass labels in the original training and validation sets. Subsequently, subclasses were grouped under their parent class. Therefore, while accuracy between subclasses may be low, as long as samples are in the same parent class, the sample will be correctly identified. For example, if static is classified as an ambiance , the sample is still correctly classified as it is ultimately labeled as “not name.” Upon generating this modified confusion matrix based on parent classes (Figure 2 ), the model with the highest F1 score is ultimately chosen as the highest-performing model. Figure 2: example of original and modified confusion matrix The final model begins with a reshape layer that transforms data processed by MFCC. It also includes two 1D convolution layers with 32 filters and 3 kernels. Each of these convolution layers is followed by a max-pooling layer to reduce dimensionality and a dropout layer to prevent overfitting [ 2 ] . This final model, deployed as a TinyML model, ensures an efficient solution for real-time keyword detection on a low-power device [ 11 ] . 2.3 Model Inference Lumename utilizes sliding-window aggregation, allowing recent and past data to be properly reflected [ 8 ] . Data is streamed from the microphone in 250ms packets. At inference time, four of these packets are combined to create one second samples. These one second samples are then classified using the previously created TinyML model, where every subclass contains a corresponding probability value from 0-100%. If the combined probabilities of the “name” subclasses are greater than an empirically predetermined threshold ( \approx 70\% ), a light and vibration motor are triggered. 3 Experiments 3.1 Variable Groups For positive testing, which tests how

Read the complete text