Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification
Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy · 2025
Read Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification by Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20474.txt.
Introduction
Anchan, Thomas, and Roy build three small perception modules for assistive technology. Their 2025 paper tests a modular deep learning framework inspired by Smart Eye systems. One module judges eye state, open or closed. Another reads facial expressions for emotion. A third identifies speakers from voice recordings.
Each module is a separate, lightweight neural network. A convolutional network handles eye images and drowsiness cues. A deeper convolutional network classifies expressions on the FER2013 dataset. A recurrent LSTM network identifies speakers from custom audio. Eye state detection reached ninety-three percent accuracy in testing. Expression recognition scored highest at 97.8 percent accuracy. Speaker identification reached 96.89 percent accuracy. The results validate specialized models for constrained assistive devices. Merging the modules into one real-time system remains future work. Latency and efficiency on embedded hardware are the next hurdles.
Catalog record
| CREATOR | Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy Abstract Developing comprehensive assistive technologies requires the seamless integration of visual and auditory perception. This research evaluates the feasibility of a modular architecture inspired by core functionalities of perceptive systems like ’Smart Eye.’ We propose and benchmark three independent sensing modules: a Convolutional Neural Network (CNN) for eye state detection (drowsiness/attention), a deep CNN for facial expression recognition, and a Long Short-Term Memory (LSTM) network for voice-based speaker identification. Utilizing the Eyes Image, FER2013, and customized audio datasets, our models achieved accuracies of 93.0%, 97.8%, and 96.89%, respectively. This study demonstrates that lightweight, domain-specific models can achieve high fidelity on discrete tasks, establishing a validated foundation for future real-time, multimodal integration in resource-constrained assistive devices. K eywords Assistive Technology \cdot Convolutional Neural Networks \cdot Long Short-Term Memory (LSTM) \cdot Speaker Identification \cdot Affective Computing \cdot Driver Drowsiness Detection \cdot Mel-Frequency Cepstral Coefficients (MFCC) 1 Introduction The development of intelligent systems capable of interpreting nuanced human cues and behaviours holds immense potential across a wide range of applications. Advancements in computer vision, natural language processing, and machine learning have spurred significant progress in perceptive technologies. This research project examines the core functionalities of Smart Eye through the development of individual modules that perform eye detection, facial expression identification, and voice recognition (speaker identification), thereby exploring the capabilities of human-computer interaction (HCI) through multimodal cues. Assistive technologies have been transformed by recent advancements in machine learning and artificial intelligence, presenting unprecedented opportunities to support individuals with diverse cognitive and physical capabilities [ 23 ] . Despite significant technological progress, persistent challenges remain in developing comprehensive, adaptive systems that can effectively interpret complex human behavioural cues. Existing assistive technologies often struggle with limited sensory interpretation, fragmented interaction mechanisms, and insufficient contextual understanding, creating substantial barriers to seamless systems interaction [ 16 ] . Technologies similar to Smart Eye have emerged as pioneering platforms in the realm of perceptive computing, leveraging advanced machine learning algorithms to decode human behavioural signals. Iterations of systems focused on discrete functionalities like gaze tracking or basic facial recognition. This methodology offers a sophisticated and responsive assistive technology ecosystem that may be integrated to various domains including automotive, healthcare, retail, and HCI. The potential societal impact of such technologies extends far beyond individual assistance. By democratising technological interaction, these systems can create more inclusive environments in healthcare, education, workplace accommodations, and personal support mechanisms. Individuals with communication challenges, motor disabilities, or neurological differences can gain unprecedented levels of autonomy and social integration [ 13 ] . Accurately and efficiently detecting the state of the eyes (open or closed) in images and video streams is a fundamental task in various domains [ 22 ] . Determining eye states can be applied in driver drowsiness systems, gaze tracking, and attention monitoring [ 10 ] . This project undertakes the development of a robust eye detection module utilising image classification techniques. We hypothesise that a trained machine learning model, such as a Convolutional Neural Network (CNN), can reliably classify eye states with high accuracy, even under potential variations in image conditions [ 21 ] . Facial expressions serve as a rich source of nonverbal communication, conveying emotions and underlying intentions and is also an essential component of HCI is the recognition of facial expressions. Real-time facial expression recognition systems find applications in domains like affective computing, human-robot interaction, and social behaviour analysis [ 18 ] . This project delves into the design and implementation of an expression identification module. We explore the effectiveness of deep learning in extracting salient features from facial images and classifying them into discrete categories [ 3 ] . Voice recognition, specifically speaker identification, focuses on distinguishing individuals based on unique characteristics within their voice patterns. This technology holds applications in security and authentication systems, personalised interfaces, and speaker diarization. This project examines the construction of a voice recognition module capable of differentiating between speakers. Our approach investigates the use of spectral features like Mel-Frequency Cepstral Coefficients (MFCCs) coupled with robust machine learning models, such as Recurrent Neural Networks (RNNs), to capture speaker information encoded in audio signals [ 14 ] . While this study focuses on the independent optimization of sensory modules, the overarching architectural vision is a cascade system. In this proposed framework, the Voice Recognition module acts as an authentication gate, activating the system only for authorized users. Subsequently, the Eye Detection module continuously monitors user attention/fatigue state, serving as a primary safety trigger. Finally, the Facial Expression module provides contextual emotional data to refine Human-Computer Interaction (HCI) responses. Validating the individual performance of these components is the critical first step toward this unified ecosystem. 2 Methodology To achieve these functionalities, the project employed a systematic methodology founded upon established machine learning paradigms. The following sections detail the specific datasets, model architectures, and evaluation strategies implemented for each component. 2.1 Eye Detection The study utilised the "Eyes Image Dataset for Machine Learning" procured from Kaggle. The dataset was partitioned into training and testing sets. Image augmentation techniques (rotation, shearing, and zooming) were applied to the training set to enhance model robustness and reduce overfitting. Images were explicitly labelled as "open" or "closed" to signify the target states for the classification task. A custom CNN architecture was designed to learn hierarchical feature representations from the eye image data. Key layers included convolutional layers for spatial feature extraction, max-pooling layers for dimensionality reduction, and dense layers for classification. Dropout and regularisation were employed to mitigate overfitting. Additionally, the InceptionV3 model, pre-trained on the ImageNet dataset, was incorporated using a transfer learning approach. Its weights were fine-tuned on the eye image dataset to leverage the rich features learned on a larger dataset and expedite training [ 1 ] . Figure 1: Visualisation of the eye detection model. The Adam optimiser with adaptive learning rate adjustments was employed for efficient training and convergence. Binary cross-entropy loss was used to optimise the classifier’s ability to discriminate between open and closed eyes. Model performance was evaluated based on categorical accuracy, precision, recall, and F1-score. Additionally, confusion matrices were generated to visualise correct and incorrect classifications. 2.2 Facial Expression Identification The dataset used for training and testing is the Facial Expression Recognition 2013 (FER2013) dataset, which contains facial images labelled with seven different emotions: anger, disgust, fear, happy, sad, surprise, and neutral [ 28 ] . The CSV file containing the pixel values and labels for facial image data is read in order to pre-process the dataset. After the pixel values are normalised to fall between [0, 1], photos are resized to fit the CNN model’s required input dimensions. Convolutional layers are placed before max-pooling layers in the CNN model architecture in order to extract features. Training stability is increased, and overfitting is avoided by using batch normalisation and dropout layers [ 2 ] . Completing the stack are fully connected layers with ReLU activation and a multi-class classification softmax layer. Figure 2: Visualisation of the facial expression model. The Adam optimiser and categorical cross-entropy loss function are used to train the model using the training set [ 12 ] . The model’s