Hierarchical human activity recognition with fusion of audio and multiple inertial sensor modalities Authors: Tuğce Alara Yılmaz, Hakan Yekta Yatbaz, Enver Ever, Adnan Yazici Published: 2025 ABSTRACT In everyday life, individuals engage in a multitude of activities, and recent technological advancements have facilitated the development of Artificial Intelligence systems that analyse these activities through data in various contexts. Human activity recognition, an essential Artificial Intelligence application in healthcare, detects deviations from normal activities, such as falls, which may indicate health issues. The widespread adoption of mobile and wearable technology enables the development of personalized activity recognition solutions. Given the critical importance of these Artificial Intelligence applications in healthcare, designing systems with high recognition accuracy is imperative. Moreover, these systems must be lightweight to ensure they operate seamlessly on end-user devices like smartphones without compromising their primary functions. Our research introduces innovative input representations and advanced methods in data fusion and multimodal learning that surpass previous methods in recognition accuracy while reducing computational and memory demands. We propose a streamlined neural network model that creatively integrates inertial and audio sensor data to generate color-coded image representations. This implemented Artificial Intelligence system was tested using a complex, publicly available activity recognition dataset organized hierarchically. Compared to earlier studies, our solution demonstrates significant performance enhancements, achieving a 91% balanced accuracy rate in activity recognition and offering substantial improvements in Central Processing Unit and memory efficiency. Introduction In exploring the diverse range of human behaviors, there lies a potential for technological advancements to capitalize on these patterns in developing supportive systems. The effectiveness of these systems hinges on their ability to accurately and autonomously identify human activities. Human Activity Recognition (HAR) applications are increasingly used in diverse areas, including healthcare1–3, sports4 smart homes5, content-based video analysis6, and surveillance7. The automation of activity recognition enhances overall quality. Many solutions have been proposed for HAR, utilizing sensors like inertial sensors, medical sensors (e.g., electrocardiographs), and multimedia sensors (cameras, microphones, etc.). Despite the extensive research in HAR, challenges remain in achieving high accuracy, mobility, discretion, and adapting to complex, everyday activities. Furthermore, HAR systems should recognize context autonomously while respecting user privacy, operating without user interaction or imparting a sense of surveillance. As mobile devices are integral to daily life, HAR modules should be compatible with these devices and designed to be lightweight to conserve battery life and not impede the primary device’s functions. Our research addresses these challenges by presenting a compact model that integrates a unique input representation and introduces new techniques in combining and learning from multiple modes, specifically focusing on inertial and audio data. This model hierarchically recognizes human activities using sensors in mobile devices. We introduce novel image-based representations for inertial and audio sensors, employing early fusion. The color-coding approach for inertial sensors follows the method in8, while various techniques, such as Mel Spectrogram or Log Mel Spectrogram (LMSPC) images, are explored for audio data. The study also examines the individual contributions of audio and inertial sensors to activity recognition. We used the public, complex ExtraSensory dataset9 for evaluation, designed for realistic multi-label HAR scenarios. To ensure a fair comparison with existing literature and address the multi-label nature of the dataset, we adopted a hierarchical representation of the dataset’s activity labels, similar to10,11. Our experiments demonstrate that our model surpasses current state-of-the-art accuracy with a lightweight framework regarding the number of parameters and memory required. In this study, we present several key contributions: first, we introduce a novel multi-modal learning method with an early fusion strategy that effectively integrates audio and sensory data within a lightweight framework, optimizing both parameter count and memory usage. Second, we develop a dual-strategy evaluation framework for HAR to assess performance across uniform and hierarchical activity classes. Our model achieves 90% accuracy for user-specified labels at the same hierarchical level, surpasses existing models by 4% to 7% in complex multiclass classification tasks, and reaches an average accuracy of 85% in scenarios where smartphone placement is critical. Lastly, we validate the lightweight nature of our approach by comparing it with other models based on floating point operations (FLOPs), parameters, and inference time. The structure of the remainder of this paper is as follows. “Related work” section discusses the state-of-the-art research in human activity recognition with mobile devices. “Proposed framework” section elaborates on the proposed model and its design specifics. “Performance evaluation” section outlines the experimental setup and findings for activity recognition through inertial and acoustic sensors. In “Discussion” section, we analyze our results and compare them with the current body of research. The paper concludes with “Conclusion” section. Related work Research in HAR can be categorized as sensor-based and vision-based, depending on the data sources they utilize. Table 1 provides an overview of commonly used public HAR datasets, primarily sourced from embedded smartphone and smartwatch sensors.Table 1Public datasets.ReferenceDatasetFrequency#Subject#ActivitySensor(s)9ExtraSensory40 Hz6051A,G,M,Aud12Opportunity30 Hz416A,G,M,O,AM13UCI-HAR50 Hz306A, G14MHealth50 Hz1012A,G,M,ECG15USC-HAD100 Hz1412A,G16WISDM20 Hz296A17PAMAP2100 Hz923A,G,M18REALDISP50 Hz1733A,G,M,4DQ19DSADS25 Hz819A,G,M20HHAR100Hz96A,G Table 1Public datasets.ReferenceDatasetFrequency#Subject#ActivitySensor(s)9ExtraSensory40 Hz6051A,G,M,Aud12Opportunity30 Hz416A,G,M,O,AM13UCI-HAR50 Hz306A, G14MHealth50 Hz1012A,G,M,ECG15USC-HAD100 Hz1412A,G16WISDM20 Hz296A17PAMAP2100 Hz923A,G,M18REALDISP50 Hz1733A,G,M,4DQ19DSADS25 Hz819A,G,M20HHAR100Hz96A,G Public datasets. The study heavily relies on the ExtraSensory dataset9, which encompasses recordings of 60 participants engaged in everyday activities, free from external monitoring. This dataset offers a comprehensive array of sensor data, including accelerometers, gyroscopes, magnetometers, audio, location, and phone usage statistics. These readings are transformed into 225 unique features through feature engineering, focusing on the mean values across three dimensions and 13 Mel Frequency Cepstral Coefficients (MFCC) channels extracted from inertial sensor data using both time and frequency domain analysis. The ExtraSensory dataset’s design reflects the diversity and unpredictability of human behavior, without predefined activity durations, acknowledging individual variations. With 51 multi-label annotations, it captures the contextual nature of activities hierarchically categorized into physical activities (h1), contextual information (h2), and phone status (h3). In this study, we use the representations

,

,

when defining new contexts. Given the possibility of simultaneous activities, a hierarchical categorization is imperative. To address the multi-label problem while preserving labelled information, a new format expressing combinations with single labels is employed. The chosen activities are based on their contribution range across the dataset and their mutual exclusivity with other user-defined activities at the same hierarchy level. Sensor based HAR studies Sensor-based HAR, applied in domains like smart homes, healthcare, and elderly monitoring, capitalizes on the expansion of wireless sensor networks and the influx from wearable, object, and environmental sensors21,22. These applications are summarized in Table 2. Employing the ExtraSensory dataset, a study9 conducted HAR for 25 behavioral contexts using logistic regression, focusing on balanced accuracy due to imbalanced activity data. It achieved 77% balanced accuracy with 5-fold and 78% with Leave-One-Out (LOO) cross-validation. Another research23 explored feature selection in elderly care with wearable and ambient sensors, obtaining a 98% accuracy using a Support Vector Machine (SVM) with a hierarchical feature selection. Convolutional Neural Networks (CNNs) are frequently utilized in HAR; for instance, Mario24 applied a three-layer CNN to waist-mounted accelerometer data, generating monochrome and colored images with F1 scores up to 94%. A shallow CNN on the University of California, Irvine (UCI) Machine Learning Repository dataset25 realized 97% accuracy with 10-fold validation. Studies26 and 27 investigated CNN kernel configurations and introduced a Seq2Dense U-Net model, respectively. The Inception network28 achieved up to 95% F1 scores across various datasets. More recently, a multimodal Transformer approach integrates sensor-fusion with self-attention and a global–temporal attention head. It achieves accuracy up to 0.919 and macro-F1 up to 0.914 using longer temporal windows on ExtraSensory (primary activities)29.Table 2Sensor-based HAR studies.StudyDataset(s)#ActivityOverlap(%)WS(s)CMValidationMetricResult9ExtraSensory25NANALR5-FoldLOOBA77% (F)78% (L)23Collected175012.8SVM10-FoldA0.9824RWHAR5505CNN5-Fold (F)LOO (L)F194% (F/M)93% (F/C)75% (L/M)76% (L/C)25UCI-HARWISDM66NA102.56CNN 10-FoldHoldoutA97%93%26UCI-HARUniMib SHARWISDMPAMAP2OpportunityWeakly Labeled617618184 5050957850502.563105.12140.96CNNHoldoutA97%76%98%93%93%92%30Collected10NA10CNNHoldoutF187%31WISDM6503.2CNN7-FoldA (A)F1 (F)95% (A)94% (F)32Ubicomp08(U)Opportunity(O)34(US)4(UC)4(OS)5(OC)NA 20 (US)300 (UC)0.5 (OS)10 (OC)CNNLSTMLODOHoldoutF190% (US)90% (UC)92%(OS)83%(OC)33UCI-HAROpportunityUniMib SHARPAMAP2WISDM61717186508stp2stp5020stp2.5664rpw151rpw2.56200rpwCNNHoldoutA(A)F1(F)96% (A)80% (F)78% (A)92% (A)98% (A)27SanitationUCI HARUCI HAPT768NANACNN/U-NetHoldoutF190.11%93.79%94.060%28OpportunityPAMAP2UCI-HAR18186NA050NA5.122.56I-NNHoldoutF195%94%95%34MHealth12502.56DBNHoldoutA97%35Collected6502.56SLFN(S)LSTM(L)HoldoutA96% (S)97% (L) Table 2Sensor-based HAR studies.StudyDataset(s)#ActivityOverlap(%)WS(s)CMValidationMetricResult9ExtraSensory25NANALR5-FoldLOOBA77% (F)78% (L)23Collected175012.8SVM10-FoldA0.9824RWHAR5505CNN5-Fold (F)LOO (L)F194% (F/M)93% (F/C)75% (L/M)76% (L/C)25UCI-HARWISDM66NA102.56CNN 10-FoldHoldoutA97%93%26UCI-HARUniMib SHARWISDMPAMAP2OpportunityWeakly Labeled617618184 5050957850502.563105.12140.96CNNHoldoutA97%76%98%93%93%92%30Collected10NA10CNNHoldoutF187%31WISDM6503.2CNN7-FoldA (A)F1 (F)95% (A)94% (F)32Ubicomp08(U)Opportunity(O)34(US)4(UC)4(OS)5(OC)NA 20 (US)300 (UC)0.5 (OS)10 (OC)CNNLSTMLODOHoldoutF190% (US)90% (UC)92%(OS)83%(OC)33UCI-HAROpportunityUniMib SHARPAMAP2WISDM61717186508stp2stp5020stp2.5664rpw151rpw2.56200rpwCNNHoldoutA(A)F1(F)96% (A)80% (F)78% (A)92% (A)98% (A)27SanitationUCI HARUCI HAPT768NANACNN/U-NetHoldoutF190.11%93.79%94.060%28OpportunityPAMAP2UCI-HAR18186NA050NA5.122.56I-NNHoldoutF195%94%95%34MHealth12502.56DBNHoldoutA97%35Collected6502.56SLFN(S)LSTM(L)HoldoutA96% (S)97% (L) Sensor-based HAR studies. 5-Fold LOO 77% (F) 78% (L) 5-Fold (F) LOO (L) 94% (F/M) 93% (F/C) 75% (L/M) 76% (L/C) UCI-HAR WISDM 6 6 10 2.56 10-Fold Holdout 97% 93% UCI-HAR UniMib SHAR WISDM PAMAP2 Opportunity Weakly Labeled 6 17 6 18 18 4 50 50 95 78 50 50 2.56 3 10 5.12 1 40.96 97% 76% 98% 93% 93% 92% A (A) F1 (F) 95% (A) 94% (F) Ubicomp08(U) Opportunity(O) 34(US) 4(UC) 4(OS) 5(OC) 20 (US) 300 (UC) 0.5 (OS) 10 (OC) CNN LSTM LODO Holdout 90% (US) 90% (UC) 92%(OS) 83%(OC) UCI-HAR Opportunity UniMib SHAR PAMAP2 WISDM 6 17 17 18 6 50 8stp 2stp 50 20stp 2.56 64rpw 151rpw 2.56 200rpw A(A) F1(F) 96% (A) 80% (F) 78% (A) 92% (A) 98% (A) Sanitation UCI HAR UCI HAPT 7 6 8 90.11% 93.79% 94.060% Opportunity PAMAP2 UCI-HAR 18 18 6 NA 0 50 NA 5.12 2.56 95% 94% 95% SLFN(S) LSTM(L) 96% (S) 97% (L) Beyond the above CNN-based models, recent studies also evaluate on ExtraSensory. A multiscale cross-modal interactive fusion network (MCIFN) extracts per-modality multiscale features, fuses them via bidirectional cross-modal attention. It refines the fused representation with a hybrid attention block. The study reports performance using balanced accuracy, recall, and specificity due to label imbalance, and does not predict context labels36. In another work37, a deep polynomial neural network (DPNN) is proposed with separate single-label classifiers for locomotion and for localization. It achieves accuracy of approximately 0.92 for both tasks on ExtraSensory dataset. Finally, a dynamic context-aware system (DCSHAR) incrementally integrates inertial signals with selected contextual inputs for activity-only recognition; on a 7-class ExtraSensory subset. It reports 0.9148 accuracy and 0.7548 Macro-F1 while reducing execution time relative to a static variant38. For efficiency, Teng et al.33 focused on lightweight CNNs, enhancing training with local loss, achieving 98% accuracy on the Wireless Sensor Data Mining (WISDM) dataset. To address limited training data, Huang et al.31 proposed two-stage CNNs, improving activity recognition through data augmentation. Activity complexity categorization in HAR influences labelling and processing, as demonstrated in32. Activities are classified as simple or complex, with complex activities first analyzed by a CNN for feature extraction and then simple activities recognized by an long short term memory (LSTM) model. Table 2 differentiates datasets and activity complexities. A deep belief network (DBN) in34 examined the impact of hidden layer configurations. The study35 compared time and frequency domain features, reporting 96% and 97% accuracy for single-layer feedforward neural network (SFLN) and LSTM models. Audio data is also integral to HAR. Jung and Chi30 utilized audio data, extracting melspectrograms for experiments, achieving an overall F1 score of 87%, with Talking recognized at 93%. Additionally, Stork et al.39 applied classical machine learning (ML) algorithms to audio data, with an ensemble voting method reaching a 92% F1 score. Vision-based HAR studies Video-based human activity, behavior, or mimic-gesture recognition has been extensively explored in research like21, 40–42 focusing on using the rich visual data from cameras for effective high-level interpretation. Table 3 summarizes these vision-based HAR studies.Table 3Vision-based HAR studies.StudyDataset(s)#ActivityClassification modelValidationMetricResult43UTD-MHAD(U)SBU-KI(S)CAD-60(C)27 (U)8 (S)12 (C)CNN 5-FoldA95%(U)96%(S)93%(C)44Collected(C)NTU RGB+D(N)140 (C)60 (N)RNN Holdout (C)NA (N)A89%(C)83%(N)45MSR Action Pairs12AENAA86%46UCF101(U)HMDB51(H)Hollywood2(HW)UCF50(UC)YouTube(Y)101 (U)51 (H)12 (HW)50 (UC)11 (Y)CNN &LSTMHoldoutA94%(U)72%(H)69%(HW)95%(UC)96%(Y)47MSR Action3D (A)MSR Daily Activity3D (D)20 (A)16 (D)HMMLOSOA93%(A)94%(D)48G3D (G)SYSU 3D HOI (S)UTD-MHAD (U)MSR Action3D (A)MSR Daily Activity3D (D)20 (G)12 (S)27 (U)20 (A)16 (D)CNNLNSOA96%(G)95%(S)89%(U)100%(A)97%(D)49MSR Action3D (M)Berkeley MHAD (B)HDM05 (H)20 (M)11 (B)65 (H)RNNLNSOA95% (M)100% (B)97% (H)*A: accuracy, BA: balanced accuracy, NA: not applicable. Table 3Vision-based HAR studies.StudyDataset(s)#ActivityClassification modelValidationMetricResult43UTD-MHAD(U)SBU-KI(S)CAD-60(C)27 (U)8 (S)12 (C)CNN 5-FoldA95%(U)96%(S)93%(C)44Collected(C)NTU RGB+D(N)140 (C)60 (N)RNN Holdout (C)NA (N)A89%(C)83%(N)45MSR Action Pairs12AENAA86%46UCF101(U)HMDB51(H)Hollywood2(HW)UCF50(UC)YouTube(Y)101 (U)51 (H)12 (HW)50 (UC)11 (Y)CNN &LSTMHoldoutA94%(U)72%(H)69%(HW)95%(UC)96%(Y)47MSR Action3D (A)MSR Daily Activity3D (D)20 (A)16 (D)HMMLOSOA93%(A)94%(D)48G3D (G)SYSU 3D HOI (S)UTD-MHAD (U)MSR Action3D (A)MSR Daily Activity3D (D)20 (G)12 (S)27 (U)20 (A)16 (D)CNNLNSOA96%(G)95%(S)89%(U)100%(A)97%(D)49MSR Action3D (M)Berkeley MHAD (B)HDM05 (H)20 (M)11 (B)65 (H)RNNLNSOA95% (M)100% (B)97% (H)*A: accuracy, BA: balanced accuracy, NA: not applicable. Vision-based HAR studies. UTD-MHAD(U) SBU-KI(S) CAD-60(C) 27 (U) 8 (S) 12 (C) 95%(U) 96%(S) 93%(C) Collected(C) NTU RGB+D(N) 140 (C) 60 (N) Holdout (C) NA (N) 89%(C) 83%(N) UCF101(U) HMDB51(H) Hollywood2(HW) UCF50(UC) YouTube(Y) 101 (U) 51 (H) 12 (HW) 50 (UC) 11 (Y) CNN & LSTM 94%(U) 72%(H) 69%(HW) 95%(UC) 96%(Y) MSR Action3D (A) MSR Daily Activity3D (D) 20 (A) 16 (D) 93%(A) 94%(D) G3D (G) SYSU 3D HOI (S) UTD-MHAD (U) MSR Action3D (A) MSR Daily Activity3D (D) 20 (G) 12 (S) 27 (U) 20 (A) 16 (D) 96%(G) 95%(S) 89%(U) 100%(A) 97%(D) MSR Action3D (M) Berkeley MHAD (B) HDM05 (H) 20 (M) 11 (B) 65 (H) 95% (M) 100% (B) 97% (H) *A: accuracy, BA: balanced accuracy, NA: not applicable. The study in43 implemented a three-level CNN using RGB, depth, and skeletal data, achieving over 93% accuracy on various datasets with 5-fold validation. Li et al.44 introduced a novel recurrent neural network (RNN) tree framework, reaching 89% accuracy in fine-grained skeleton-based recognition. Jia et al.45 effectively used noisy data, employing a three-layered sparse autoencoder with denoising capability, achieving best results with 20% noisy data, leading to 86% accuracy. In46, the introduced framework and methodology surpassed existing methods. Another study, Jalal47, using red green blue-depth (RGBD) data and leave-one-subject-out cross-validation, reported up to 94% accuracy with a hidden Markov model (HMM). Wang et al.48 presented an RGB-D video-based action recognition method using bidirectional rank pooling and dynamic depth images, alongside CNN for enhanced recognition. The skeletal joint vision-based study in49 adapted an RNN model for classification, reporting up to 100% accuracy for the Berkeley Multimodal Human Action Database (MHAD) dataset of 7 subjects in Table 3. Overall, these studies demonstrate the variety and advancement in vision-based HAR, utilizing convolutional and recurrent networks to process intricate visual datasets for accurate activity classification. Audio processing for HAR Mel Frequency Cepstral Coefficients (MFCCs) play a crucial role in audio signal processing for HAR, derived through a sequence of signal processing steps. Initially, audio waveforms are segmented into frames with a Discrete Fourier Transform (DFT) applied to convert these into frequency domain representations. The resulting Power Spectrum, indicating frequency magnitudes, is log-scaled to create the Log Power Spectrum (Log-PS), highlighting harmonic content. To convert frequencies into the mel scale, a suitable number of mel bands are chosen, typically between 32 and 64. These mel filter banks transform the Log-PS into the Mel Spectrogram (MSPC) through matrix multiplication, illustrating frequency variations over time. Subsequently, a Discrete Cosine Transform (DCT) condenses the Mel Spectrogram into a set of real-valued cepstral coefficients, forming the MFCC matrix, which usually consists of 12 to 13 coefficients. In our study, we employ Mel Spectrograms, which are preferred for deep learning-based audio analysis and MFCCs which are generally used for conventional machine learning techniques in HAR. Also, we employ Log Melspectrograms during performance evaluation. The choice between Mel Spectrograms and MFCCs depends on the specific requirements of the employed learning method. Initially, we applied mel-scaling to the waveform signal, converting it into a linear form known as the Mel Spectrogram. We then took the logarithm of the Power Spectrum (PS) to produce the Log Mel Spectrogram, which captures continuous frequencies with harmonic components. Finally, we used the DCT to transform the Log Mel Spectrogram into a matrix of real-valued cepstral coefficients, resulting in MFCCs. This process is crucial for extracting meaningful features from audio signals for subsequent analysis. Multi-label classification In HAR, multi-label classification captures simultaneous activities effectively. The9 study, utilizing the ExtraSensory dataset, gathered data via smartphone and smartwatch sensors from 60 participants across 5000 hours. This study explored both single-sensor classifiers and sensor fusion techniques. The highest single-sensor accuracy (0.76) was observed with phone state sensors, while early fusion in sensor fusion reached 0.86 accuracy in leave-one-out (LOO) experiments. Audio sensors led to balanced accuracy for single sensors (0.70), and both averaging and learned weight methods in late fusion achieved 0.80. The50 study also used the ExtraSensory dataset, implementing early and late sensor fusion, along with a multiple-layer perceptron (MLP) for context recognition, achieving a balanced accuracy of 0.762 with five-fold cross-validation. Research in51 applied boosting algorithms to HAR, outperforming Neural Networks, SVMs, MLP, and logistic regression-based methods with an F1 score of approximately 0.88. In52, a pre-trained CNN feature extractor was used, achieving F-scores of 0.5277 and 0.2142 for inertial and audio sensors, respectively, with five-fold cross-validation. Multi-label classification in HAR was explored in10, focusing on coarse and fine-grained levels. Their Coarse-to-Fine Human Activity Recognition (C2FHAR) approach, integrating Decision Tree, Random Forest, and Neural Networks, achieved up to 0.83 and 0.86 balanced physical activities and phone placement accuracies. The Context-aware human activity recognition (CAHAR) framework in11 combined behavioural context with physical activities, using classifiers like random forest and K-nearest neighbor (kNN), achieving average balanced accuracies of 0.80 for physical activities and 0.77 for behavioural context with 10-fold cross-validation. Ehatisham-ul Haq and Azam53 presented another scheme using classifiers like Decision Tree and kNN, obtaining balanced accuracies up to 0.83 with 5-fold cross-validation. Alternatively in54, active learning strategies are investigated for HAR, achieving a balanced accuracy of around 0.79 with less labeled data, comparable to supervised methods. Finally, in recent attempts to tackle multi-label prediction for CA-HAR, a deep heterogeneous contrastive hyper-graph method (DHC-HGL) is proposed to model inter-label structure across activity and placement using contrastive objectives55. Authors present results with MCC and Macro-F1, including activity-category averages around Matthews Correlation Coefficient (MCC) of 0.855 and 0.923 Macro-F1 on ExtraSensory Dataset. In addition, a label-semantics method (SEAL) encodes activity and placement labels with semantic representations to guide joint prediction is proposed in56, which achieves Activity MCC of 0.922, 0.959 Macro-F1, Context MCC of 0.977 and 0.988 Macro-F1. In this research, our methods excel in accuracy, both in single hierarchy classification without grouping and in multiclass classification involving intricate activity-context pairings, while demanding less memory and computational resources for real-time activity monitoring. Proposed framework In this section, we detail our proposed framework, focusing on aspects such as dataset selection, types of inputs, the process of data preprocessing, and the implementation of an efficient, lightweight neural network for hierarchical classification. Figure 1 provides a visual representation of the framework, and the following sections will explore each element in greater depth. While the framework is demonstrated using the ExtraSensory dataset, its design allows for easy adaptation to various other datasets.Fig. 1A concise framework for activity recognition using sensor data. This process includes preprocessing including a sliding window with 50% overlap to create classification samples. Activities and contexts are identified through user and activity clustering with a hierarchical labelling approach. Sensor data normalization adjusts for hardware-specific ranges, ensuring uniform interpretation across sensors. Color-coded images for inertial and audio data are merged for multi-modal recognition, leading to a comprehensive recognition mode of Activity-Context-Placement. A concise framework for activity recognition using sensor data. This process includes preprocessing including a sliding window with 50% overlap to create classification samples. Activities and contexts are identified through user and activity clustering with a hierarchical labelling approach. Sensor data normalization adjusts for hardware-specific ranges, ensuring uniform interpretation across sensors. Color-coded images for inertial and audio data are merged for multi-modal recognition, leading to a comprehensive recognition mode of Activity-Context-Placement. Dataset description and pre-processing techniques In our study, we selected the9 dataset for its comprehensive coverage of activities, contexts, phone positions, and multi-label annotations. This dataset, comprising multi-modal sensory data from 60 users, was ideal due to its extensive participant range and detailed contextual information. Unique to this dataset is the inclusion of audio information, which enhances HAR systems. For instance, in-home settings, audio signals can help identify specific activities such as “Talking” or “Watching TV”. The dataset comprises unprocessed and processed inertial sensor data and statistical features extracted from it. Raw audio data is omitted for privacy reasons, but the dataset includes MFCCs as a numerical audio representation. Additionally, it features 26 extra audio-derived characteristics from smartphones, encompassing the mean of the first 13 MFCC channels and the standard deviations of the remaining 13, averaging over 400 frames. The dataset faces imbalance issues due to varying user contributions and uncontrolled collection settings. To mitigate this, we use a sliding window method with 20 data points per sample and 50% overlap between windows, aligning with established practices. This approach helps balance the dataset, especially for minority classes and ensures uniform input feature lengths. We also cluster feature data by user and activity. Labels are applied to activities with hierarchical definitions at each hierarchy level. Data is segmented by activity or context per user, then grouped by sensor readings, forming clusters for each activity and individual. Following clustering, windowing is applied to create images. Each sensor in our dataset displays hardware-specific distribution ranges, with variations for different activities. Typically, activities show consistent patterns: minor sensory changes indicate low-effort activities, while significant fluctuations suggest high-effort actions. Based on this, we first calculate the standard deviation for each sensor channel (see equation 1). Then, we normalize the data to unit variance as detailed in Equation 2, with i and j representing the element and sensor channel, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{{\overline{x}}}$$\end{document} denoting the mean. This user-specific normalization accounts for variations in sensor outputs due to different hardware and software. This crucial step is completed before categorizing activities for each individual.1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbf{s(x)}= & \mathbf{\sqrt{\frac{1}{N-1} \sum _{i=1}^N (x_i - \overline{x})^2}} \end{aligned}$$\end{document}2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbf{x_{ij}}= & \mathbf{\frac{x_{ij} - \overline{x_j}}{s(x_j)}} \end{aligned}$$\end{document} Image generation from sensor data This study addresses the challenge of visualizing actions or contexts in HAR systems by converting activities into “activity images.” These RGB images visually represent daily activities or contexts, aiding the model more efficiently extracting activity recognition patterns. In our approach, each pixel’s numerical value, typically in the [0, 255] range, is derived from sensor data after normalization. This data is then mapped to pixel values, with individual adjustments for each person and sensor channel, preserving natural behavior patterns. We map the X, Y, and Z channels to RGB pixel channels for inertial sensors (like smartphone accelerometers, gyroscopes, magnetometers, and smartwatch accelerometers). This technique, inspired by the study in8, uses four inertial sensors represented in an image row, as shown in Fig. 2a for the activity “Lying Down”. Audio sensors, however, differ as MFCC sequences and Mel-scaled Power Cepstral Coefficients (MSPCs)/Log-Mel Power Spectral Coefficients (LMPSCs) have more channels. We represent these as grayscale images, later converting them to RGB for early fusion by assigning each grayscale value to all R, G, and B channels. This method is demonstrated in Fig. 2b–d.Fig. 2Inertial and audio sensory fusion image: this image merges inertial sensor data and audio features. The first subfigure shows inertial sensors (accelerometer, gyroscope, magnetometer), each row depicting a different sensor. The following subfigures display audio features (MFCC, MSPC, LMSPC), each row representing a specific channel. Columns across subfigures indicate different timesteps. Inertial and audio sensory fusion image: this image merges inertial sensor data and audio features. The first subfigure shows inertial sensors (accelerometer, gyroscope, magnetometer), each row depicting a different sensor. The following subfigures display audio features (MFCC, MSPC, LMSPC), each row representing a specific channel. Columns across subfigures indicate different timesteps. Proposed lightweight model To train a HAR classification model, we use fused color-coded sensor data images. Our proposed CNN architecture includes 4 convolutional layers, each with 32 output channels and a 3x3 kernel. Following each convolutional layer is a 1x2 max pooling layer, and all these layers use the Rectified Linear Unit (ReLU) activation function. The network then includes a flattened layer to reshape feature dimensions before a fully connected layer with 512 hidden units, followed by another with 256 hidden units. The final layer, matching the number of classes, uses a softmax activation function suitable for multi-class issues. We employ stochastic gradient descent (SGD) as an optimizer with a learning rate of 0.0001 and a momentum of 0.9. The loss function is categorical cross-entropy Loss, and we train for 100 epochs. An early stop condition is implemented to halt training when validation loss no longer decreases, optimizing for robust predictions. The parameters of CNN were chosen through a meticulous process of hyperparameter tuning. This involved experimenting with various configurations to identify the optimal settings that enhance the model’s performance. Key parameters such as learning rate, batch size, number of epochs, and the architecture of the CNN layers were adjusted iteratively. By evaluating the model’s accuracy and loss metrics on validation data, these parameters are fine-tuned to achieve the best possible recognition performance. Performance evaluation In this section, we outline the experimental setup of our study and discuss the performance results, taking into account both dependent and independent scenarios for the main group. Experimental setup and performance metrics We evaluate the effectiveness of the proposed lightweight model using 5-fold cross-validation, reporting the average outcomes from all folds. The experiments cover both combined and individual sensor setups, and their details are provided in this section. The Extrasensory dataset, consisting of genuine real-world samples, is imbalanced. To address this, we implement a class weight strategy to amplify the impact of less represented activities during training. This approach balances the influence of each class on gradient updates, reducing the model’s tendency to favor more prevalent classes. The weights for the c-th class can be calculated as3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W(c) = \frac{n_S}{n_{C}\cdot n_{c,s}}, \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_S$$\end{document} is the total number of samples in the training set, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_C$$\end{document} is the total number of classes, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{c,s}$$\end{document} is the total number of samples in the selected class57. In our study, we employed commonly used classification metrics such as accuracy and F1-score to demonstrate and compare the model’s effectiveness. Given the imbalanced nature of our dataset, we have chosen specific adaptations of these metrics for a more accurate assessment:Accuracy (ACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {ACC} = \frac{\text {TP}+\text {TN}}{\text {TP} + \text {TN}+\text {FP}+\text {FN}} \end{aligned}$$\end{document}Balanced Accuracy (BACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {BACC} = \frac{1}{2} \left( \frac{\text {TP}}{\text {TP} + \text {FN}} + \frac{\text {TN}}{\text {TN} + \text {FP}} \right) \end{aligned}$$\end{document}Precision (PRE), Recall (REC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {PRE} = \frac{\text {TP}}{\text {TP} + \text {FP}}, \quad \text {REC} = \frac{\text {TP}}{\text {TP} + \text {FN}} \end{aligned}$$\end{document}F1 Score (F1-Type): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {F1-Type} = \frac{2 \times (P_{\text {Type}} \times R_{\text {Type}})}{P_{\text {Type}} + R_{\text {Type}}} \end{aligned}$$\end{document}Specificity (SPE): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {SPE} = \frac{\text {TN}}{\text {TN} + \text {FP}} \end{aligned}$$\end{document}Weighted F1 (W-F1): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {W-F1} = \sum _{i=1}^{n} w_i \times \text {F1}_{i} \end{aligned}$$\end{document} Accuracy (ACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {ACC} = \frac{\text {TP}+\text {TN}}{\text {TP} + \text {TN}+\text {FP}+\text {FN}} \end{aligned}$$\end{document} Accuracy (ACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {ACC} = \frac{\text {TP}+\text {TN}}{\text {TP} + \text {TN}+\text {FP}+\text {FN}} \end{aligned}$$\end{document} Balanced Accuracy (BACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {BACC} = \frac{1}{2} \left( \frac{\text {TP}}{\text {TP} + \text {FN}} + \frac{\text {TN}}{\text {TN} + \text {FP}} \right) \end{aligned}$$\end{document} Balanced Accuracy (BACC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {BACC} = \frac{1}{2} \left( \frac{\text {TP}}{\text {TP} + \text {FN}} + \frac{\text {TN}}{\text {TN} + \text {FP}} \right) \end{aligned}$$\end{document} Precision (PRE), Recall (REC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {PRE} = \frac{\text {TP}}{\text {TP} + \text {FP}}, \quad \text {REC} = \frac{\text {TP}}{\text {TP} + \text {FN}} \end{aligned}$$\end{document} Precision (PRE), Recall (REC): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {PRE} = \frac{\text {TP}}{\text {TP} + \text {FP}}, \quad \text {REC} = \frac{\text {TP}}{\text {TP} + \text {FN}} \end{aligned}$$\end{document} F1 Score (F1-Type): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {F1-Type} = \frac{2 \times (P_{\text {Type}} \times R_{\text {Type}})}{P_{\text {Type}} + R_{\text {Type}}} \end{aligned}$$\end{document} F1 Score (F1-Type): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {F1-Type} = \frac{2 \times (P_{\text {Type}} \times R_{\text {Type}})}{P_{\text {Type}} + R_{\text {Type}}} \end{aligned}$$\end{document} Specificity (SPE): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {SPE} = \frac{\text {TN}}{\text {TN} + \text {FP}} \end{aligned}$$\end{document} Specificity (SPE): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {SPE} = \frac{\text {TN}}{\text {TN} + \text {FP}} \end{aligned}$$\end{document} Weighted F1 (W-F1): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {W-F1} = \sum _{i=1}^{n} w_i \times \text {F1}_{i} \end{aligned}$$\end{document} Weighted F1 (W-F1): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \text {W-F1} = \sum _{i=1}^{n} w_i \times \text {F1}_{i} \end{aligned}$$\end{document} Performance analysis To assess model performance, we employ two complementary evaluation protocols that align with the three-level label hierarchy (H1–H3): H1 represents primary physical activities, H2 refers to placement-independent contexts (PIC), and H3 covers context–placement combinations (PDC). In the main-group-independent (MGI) setting, each hierarchy level is classified separately, without conditioning on other levels. For instance, H1 predicts six activities, H2 predicts 15 PIC classes aggregated across all activities, and H3 predicts 28 PDC classes pooled across all activity and placement combinations. In contrast, the main-group-dependent (MGD) setting conditions predictions on the main activity. Here, the model first determines the H1 activity and then classifies its related H2 contexts (four PIC groups: PIC-1 to PIC-4) and H3 context–placement pairs (four PDC groups: PDC-1 to PDC-4). Finally, per-class modality patterns are visualized in figures, while their corresponding averaged results are summarized in the tables where relevant. Independent hierarchical evaluation of activity, context, and device placement labels In this section, we test primary physical activities, placement-independent and placement-dependent contexts, as shown in Fig. 3. These are evaluated hierarchically at the same level but without grouping them based on prior activity. Each primary physical activity, context, and device placement is treated as an independent label. This means primary physical activities in Fig. 3 are assessed without considering related contexts or device placements. Likewise, placement-independent and dependent contexts are evaluated separately. This approach ensures the classification process does not assume any hierarchical connection between the activity, context, or device placement. Every label, whether an activity, a context, or a placement, is analyzed with equal importance, allowing for an unbiased assessment of the dataset. Such a method enables a detailed examination of labels individually, potentially offering deeper insights into the data beyond predefined activity groupings. Figure 3 illustrates the structure of the ExtraSensory dataset, categorizing activities into six primary categories (such as laying down, walking, cycling) and further dividing them into specific contexts (e.g., sleeping, watching TV). Additionally, the lowest level details device placements (for instance, ’In Hand’ represented as ’PIH’). In this depiction, each activity-context-placement is treated as an individual class or grouped within primary activities, emphasizing pertinent contexts and placements. Activities utilized solely in group-independent experiments are indicated with an asterisk (*), while those exclusive to group-dependent experiments are marked with a double asterisk (**).Fig. 3Hierarchical structure of activities in the extrasensory dataset. Hierarchical structure of activities in the extrasensory dataset. At the first hierarchy level, for independent testing of primary activities, six physical activities are depicted in Fig. 3. The average outcomes of 5-fold cross-validation for various sensor inputs are detailed in Table 4.Table 45-Fold results of primary physical activities (context independent).SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial76.4%0.7020.94382.3%0.6580.6770.7640.765MFCC79.8%0.7130.94983.1%0.7390.7220.7980.796MSPC79.7%0.7320.95184.2%0.7510.7380.7970.798LMSPC77.5%0.6860.94581.5%0.7200.7000.7750.773Inertial & MFCC87.4%0.8260.97089.8%0.8320.8280.8740.874Inertial & MSPC87.9%0.8530.97191.2%0.8220.8340.8790.880Inertial & LMSPC87.1%0.8060.96888.7%0.8310.8170.8710.870 Table 45-Fold results of primary physical activities (context independent).SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial76.4%0.7020.94382.3%0.6580.6770.7640.765MFCC79.8%0.7130.94983.1%0.7390.7220.7980.796MSPC79.7%0.7320.95184.2%0.7510.7380.7970.798LMSPC77.5%0.6860.94581.5%0.7200.7000.7750.773Inertial & MFCC87.4%0.8260.97089.8%0.8320.8280.8740.874Inertial & MSPC87.9%0.8530.97191.2%0.8220.8340.8790.880Inertial & LMSPC87.1%0.8060.96888.7%0.8310.8170.8710.870 5-Fold results of primary physical activities (context independent). The model achieves an 82% Balanced Accuracy Score (BACC) using only inertial sensors. However, individual sound sensors, particularly MFCC and MSPC features, outperform inertial ones, with LMSPC being slightly less effective. MSPC is the most informative audio sensor type, reaching an 84% BACC, compared to 83% with MFCC and about 82% with LMSPC. The impact of sensor fusion is evident, with inertial and MSPC sensor fusion yielding an average BACC of 91%, while other fusion combinations also show high accuracy. Figure 4 displays the average F1 scores for each fold per class, highlighting the significant role of audio sensors. The integration of auditory sensors with inertial ones notably enhances activity detection accuracy. For example, the MSPC component significantly aids in identifying activities like “Cycling,” as shown in the bar graph. It is also noted that audio sensors alone can sufficiently recognize activities for certain activities, particularly with the use of MSPC in audio.Fig. 4First hierarchical level F1 scores per activity with 5-Fold cross validation (main activity independent group). First hierarchical level F1 scores per activity with 5-Fold cross validation (main activity independent group). At the second hierarchical level, shown in Fig. 3, the classification task differentiates among 15 unique classes formed by combining activities with their related contexts. These classes are represented by concatenated labels combining activity and context abbreviations, like ’‘STD_IND’ for ’Standing Indoors’. This labelling applies to Placement Independent Contexts (PIC), indicating contexts unaffected by the device’s location. Table 5 presents results in various metrics and their averages. Using only inertial sensor data, the model achieves a 75% BACC. Auditory features, however, yield higher scores. MSPC is the most effective of the three audio types, achieving a 79% BACC. The results suggest a notable improvement when audio and inertial sensors are fused. The top performance is observed with inertial and MFCC fusion, reaching about 81% BACC. The BACC for fusions of inertial-MSPC and inertial-LMSPC are both close, each scoring around 80%. The findings indicate that combining auditory and inertial sensors improves performance in recognizing physical activities and identifying contexts. Additionally, the results highlight the efficacy of audio sensors in context recognition, irrespective of the specific type used.Table 55-fold results of placement independent contexts.SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial72.5%0.5260.97975.2%0.5150.5140.7250.729MFCC75.9%0.5520.98276.7%0.5610.5520.7590.762MSPC79.1%0.5990.98379.1%0.6150.6030.7910.790LMSPC77.3%0.5770.98278.0%0.6010.5840.7730.772Inertial & MFCC81.3%0.6260.98580.6%0.6330.6280.8130.812Inertial & MSPC80.5%0.6180.98580.2%0.6380.6240.8050.804Inertial & LMSPC80.0%0.6180.98480.1%0.6290.6170.8000.800 Table 55-fold results of placement independent contexts.SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial72.5%0.5260.97975.2%0.5150.5140.7250.729MFCC75.9%0.5520.98276.7%0.5610.5520.7590.762MSPC79.1%0.5990.98379.1%0.6150.6030.7910.790LMSPC77.3%0.5770.98278.0%0.6010.5840.7730.772Inertial & MFCC81.3%0.6260.98580.6%0.6330.6280.8130.812Inertial & MSPC80.5%0.6180.98580.2%0.6380.6240.8050.804Inertial & LMSPC80.0%0.6180.98480.1%0.6290.6170.8000.800 5-fold results of placement independent contexts. Figure 5 presents the average F1 scores for each activity-context pair. Generally, combining inertial and auditory sensors yields better results that are consistent with findings at other levels. Performance varies significantly across contexts, with some showing high accuracy and others lower, likely due to data distribution disparities. The figure also highlights the significance of audio in enhancing model performance. Activities with distinctive sounds, like “Sitting In a Car” with vehicle noise or “Standing Outside” with environmental noise, are recognized with greater accuracy.Fig. 5Second hierarchical level F1 scores per context with 5-Fold cross validation (main activity independent group). Second hierarchical level F1 scores per context with 5-Fold cross validation (main activity independent group). At the third hierarchy level, depicted in Fig. 3, a multi-class classification encompasses 28 distinct categories. This level evaluates smartphone placements for each combination of primary physical activity and context, known as Placement Dependent Contexts (PDC). The model’s performance for each input type, including results from all folds and overall outcomes, is detailed in Table 6.Table 65-Fold results of placement dependent contexts.SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial72.5%0.4420.98971.6%0.4360.4240.7250.731MFCC74.8%0.4360.99071.3%0.4480.4310.7480.750MSPC71.2%0.3840.98868.6%0.4060.3840.7120.714LMSPC68.7%0.3890.98768.8%0.3840.3760.6870.693Inertial & MFCC79.9%0.5370.99276.4%0.5610.5360.7990.801Inertial & MSPC79.3%0.5340.99276.3%0.5680.5300.7930.794Inertial & LMSPC78.2%0.5320.99176.2%0.5570.5330.7820.784 Table 65-Fold results of placement dependent contexts.SensorACCRECSPEBACCPREMac-F1Mic-F1W-F1Inertial72.5%0.4420.98971.6%0.4360.4240.7250.731MFCC74.8%0.4360.99071.3%0.4480.4310.7480.750MSPC71.2%0.3840.98868.6%0.4060.3840.7120.714LMSPC68.7%0.3890.98768.8%0.3840.3760.6870.693Inertial & MFCC79.9%0.5370.99276.4%0.5610.5360.7990.801Inertial & MSPC79.3%0.5340.99276.3%0.5680.5300.7930.794Inertial & LMSPC78.2%0.5320.99176.2%0.5570.5330.7820.784 5-Fold results of placement dependent contexts. The fusion of inertial and auditory sensor features results in improved recognition performance, as evidenced by the average outcomes from 5-fold cross-validation metrics. The model displays comparable performances across three fusion variations with different audio features, achieving an overall balanced accuracy score of 76%. Utilizing only inertial sensors yielded a balanced accuracy of 72%. Of the three audio features, MFCC emerges as the most informative for prediction, with a balanced accuracy score of 71%. Figure 6 contains two subfigures that show the model’s performance across each class for different input types, measured by the F1 score. The performance based on Placement Dependent Contexts (PDC) varies significantly across classes, with some scenarios achieving high success. This variability is primarily due to the uneven distribution of user data. Classes with more examples show better results, as the model learns their features more accurately. Conversely, the model performs poorly with classes that have fewer data samples.Fig. 6Third hierarchical level F1 scores per context-placement with 5-Fold cross validation (main activity independent group). Third hierarchical level F1 scores per context-placement with 5-Fold cross validation (main activity independent group). Hierarchical evaluation based on main group dependency Our experimental evaluation extends beyond assessing the standalone performance of our representational learning framework and the associated lightweight neural network. We delve into the hierarchical nature of the data to examine the model’s precision in-class differentiation. This additional analysis focuses on the model’s skill in identifying various contexts within a single activity, like “Sitting,” and its ability to differentiate between different device placements within these activities. This approach differs from previous sections, specifically aiming to discern context and placement labels for each primary physical activity. The goal is to underscore the advantages of using a hierarchical structure in classification. At the second hierarchy level (H2), focusing on context classification within each primary physical activity, there are four main test groups: “Laying Down” (PIC-1), “Standing” (PIC-2), “Sitting” (PIC-3), and “Watching TV” (PIC-4). The results for these groups are detailed in Table 7.Table 7Average 5-fold results of placement independent context group testing (context dependent).Test groupSensorACCRECSPEBACCPREMac-F1Mic-F1W-F1PIC-1Inertial96.6%0.7780.91084.4%0.7580.7660.9660.967MFCC97.4%0.8150.92787.1%0.8120.8130.9740.974MSPC96.9%0.7670.90383.5%0.7840.7740.9690.969LMSPC96.6%0.7290.88180.5%0.7770.7490.9660.966Inertial & MFCC98.0%0.8640.94890.6%0.8500.8560.9800.980Inertial & MSPC97.6%0.8240.92887.6%0.8290.8240.9760.976Inertial & LMSPC97.5%0.8080.92586.7%0.8160.8100.9750.975PIC-2Inertial95.6%0.7290.72972.9%0.7630.7420.9560.955MFCC96.5%0.7340.73473.4%0.8350.7690.9650.962MSPC95.3%0.7220.72272.2%0.7370.7280.9530.952LMSPC95.4%0.6860.68668.6%0.7480.7070.9540.950Inertial & MFCC96.3%0.8010.80180.1%0.8020.7990.9630.963Inertial & MSPC96.8%0.8140.81481.4%0.8330.8200.9680.967Inertial & LMSPC96.8%0.7610.76176.1%0.8480.7970.9680.965PIC-3Inertial79.8%0.7790.92485.2%0.7930.7850.7980.798MFCC80.7%0.7930.92886.1%0.8030.7970.8070.807MSPC83.3%0.8220.93587.9%0.8450.8310.8330.832LMSPC82.3%0.8180.93387.6%0.8210.8180.8230.822Inertial & MFCC87.0%0.8690.95191.0%0.8690.8690.8700.870Inertial & MSPC86.6%0.8630.94990.6%0.8750.8680.8660.866Inertial & LMSPC86.5%0.8670.94990.8%0.8690.8670.8650.865PIC-4Inertial51.5%0.4820.81965.1%0.5010.4980.5140.510MFCC53.5%0.5090.82766.8%0.5160.5090.5350.532MSPC50.9%0.4410.81863.0%0.4870.4800.5090.507LMSPC49.9%0.4210.80461.2%0.4650.4590.4990.496Inertial & MFCC55.9%0.5360.85069.3%0.5370.5320.5590.555Inertial & MSPC56.4%0.5390.84269.8%0.5410.5370.5640.560Inertial & LMSPC55.9%0.5370.84169.0%0.5380.5340.5590.556 Table 7Average 5-fold results of placement independent context group testing (context dependent).Test groupSensorACCRECSPEBACCPREMac-F1Mic-F1W-F1PIC-1Inertial96.6%0.7780.91084.4%0.7580.7660.9660.967MFCC97.4%0.8150.92787.1%0.8120.8130.9740.974MSPC96.9%0.7670.90383.5%0.7840.7740.9690.969LMSPC96.6%0.7290.88180.5%0.7770.7490.9660.966Inertial & MFCC98.0%0.8640.94890.6%0.8500.8560.9800.980Inertial & MSPC97.6%0.8240.92887.6%0.8290.8240.9760.976Inertial & LMSPC97.5%0.8080.92586.7%0.8160.8100.9750.975PIC-2Inertial95.6%0.7290.72972.9%0.7630.7420.9560.955MFCC96.5%0.7340.73473.4%0.8350.7690.9650.962MSPC95.3%0.7220.72272.2%0.7370.7280.9530.952LMSPC95.4%0.6860.68668.6%0.7480.7070.9540.950Inertial & MFCC96.3%0.8010.80180.1%0.8020.7990.9630.963Inertial & MSPC96.8%0.8140.81481.4%0.8330.8200.9680.967Inertial & LMSPC96.8%0.7610.76176.1%0.8480.7970.9680.965PIC-3Inertial79.8%0.7790.92485.2%0.7930.7850.7980.798MFCC80.7%0.7930.92886.1%0.8030.7970.8070.807MSPC83.3%0.8220.93587.9%0.8450.8310.8330.832LMSPC82.3%0.8180.93387.6%0.8210.8180.8230.822Inertial & MFCC87.0%0.8690.95191.0%0.8690.8690.8700.870Inertial & MSPC86.6%0.8630.94990.6%0.8750.8680.8660.866Inertial & LMSPC86.5%0.8670.94990.8%0.8690.8670.8650.865PIC-4Inertial51.5%0.4820.81965.1%0.5010.4980.5140.510MFCC53.5%0.5090.82766.8%0.5160.5090.5350.532MSPC50.9%0.4410.81863.0%0.4870.4800.5090.507LMSPC49.9%0.4210.80461.2%0.4650.4590.4990.496Inertial & MFCC55.9%0.5360.85069.3%0.5370.5320.5590.555Inertial & MSPC56.4%0.5390.84269.8%0.5410.5370.5640.560Inertial & LMSPC55.9%0.5370.84169.0%0.5380.5340.5590.556 Average 5-fold results of placement independent context group testing (context dependent). In the test group PIC-1, focused on “Lying down” contexts, the dataset is notably unbalanced, yet the results are encouraging. The model achieves a 90% balanced accuracy score with MFCC and inertial sensor fusion. Notably, MFCC alone shows a 3% improvement over inertial sensors alone, highlighting the impact of audio in recognition. For the PIC-2 test group, which examines “Standing” contexts, the best results come from combining inertial sensors with MSPC, reaching an 81% score. This finding underscores the role of environmental sounds in activity recognition, particularly between the two contexts of “Indoors” and “Outside.” Despite a significant imbalance in sample sizes between these contexts, the model effectively learns from this dataset by employing a class weight strategy. PIC-3 represents the “Sitting” activity contexts. The model achieves an 85% BACC with just inertial data, improving to 88% with only MSPC audio. Combining data further enhances performance, with all audio data versions reaching around 91% BACC when fused with inertial sensors. For PIC-4, representing “Walking” contexts, the model faces more challenges. The best performance is seen with an LMSPC and inertial sensor fusion, achieving a 69% BACC. The model struggles to differentiate between “Outdoors” and “Talking” activities, possibly due to confusion with external human speech. Figure 7 displays context-based F1 scores. High performance is noted for PIC-1, PIC-2, and PIC-3, with an average BACC of 87%. While audio improves results for PIC-4, its scores are lower than other groups. Overall, hierarchical grouping techniques yield marginally better outcomes than non-grouped approaches.Fig. 7PIC F1 scores per context (PIC) - 5-Fold (main activity dependent group). PIC F1 scores per context (PIC) - 5-Fold (main activity dependent group). At hierarchy level three (H3), the primary group-dependent testing mechanism for context-placement pairs involves the creation of four test groups similar to H2. Nevertheless, at this level, the user-defined labels differ in another manner. They continue to represent distinctions in activity-context divisions but now account for differences based on smartphone usage. Table 8 lists the average results of 5-fold cross-validation. The first test group, PDC-1, focuses on the “Lying down” activity, considering device placement. In this scenario, while inertial sensors alone are quite effective in differentiating context and phone placement, the fusion of audio and inertial sensors further enhances performance, boosting the BACC by approximately 2%. In the PDC-2 group, the model predicts both the context and phone placement for the primary activity “Standing.” When the audio feature MFCC is combined with inertial sensors, the overall performance reaches 83%. This combination of MFCC and inertial sensors also proves effective for the PDC-3 group, which focuses on the “Sitting” activity. In the PDC-4 group, although the inclusion of audio features enhances the overall performance, achieving a BACC of 78% is still lower than the performance in other test groups.Table 8Average 5-fold results of context dependent placement classification.Test groupSensorACCRECSPEBACCPREMac-F1Mic-F1W-F1PDC-1Inertial98.6%0.8070.95388.0%0.8110.8000.9860.987MFCC98.6%0.7870.96287.4%0.7950.7800.9860.986MSPC97.3%0.5440.89071.7%0.6010.5600.9730.974LMSPC97.4%0.5410.88471.2%0.6680.5750.9740.974Inertial & MFCC98.6%0.8240.96289.3%0.7920.8030.9860.987Inertial & MSPC98.5%0.8090.97089.9%0.8040.8060.9850.986Inertial & LMSPC98.2%0.7550.93784.6%0.7490.7400.9820.982PDC-2Inertial80.5%0.6310.94979.0%0.6700.6420.8050.805MFCC78.1%0.5920.94276.7%0.6200.5990.7810.783MSPC68.3%0.4950.91870.7%0.4960.4880.6830.683LMSPC68.7%0.4850.91870.1%0.4910.4700.6870.687Inertial & MFCC84.7%0.6840.97182.7%0.6770.6700.8470.848Inertial & MSPC82.9%0.6810.95882.0%0.7010.6800.8290.829Inertial & LMSPC83.0%0.6620.95881.0%0.7200.6750.8300.831PDC-3Inertial78.1%0.5910.96978.0%0.5900.5780.7810.782MFCC80.4%0.6290.97280.0%0.5980.5980.8040.805MSPC81.5%0.6160.97379.5%0.6130.6030.8150.815LMSPC79.6%0.6310.97180.1%0.5770.5790.7960.796Inertial & MFCC86.7%0.7000.98184.0%0.7080.6890.8670.867Inertial & MSPC85.1%0.6590.97981.9%0.6560.6460.8510.852Inertial & LMSPC85.0%0.6640.97882.1%0.6880.6670.8500.852PDC-4Inertial51.0%0.4700.92569.7%0.4460.4370.5100.511MFCC50.6%0.5210.92372.2%0.5190.5010.5060.506MSPC39.9%0.3310.90661.9%0.3450.3190.3990.400LMSPC39.0%0.3070.90460.6%0.3390.3060.3900.389Inertial & MFCC56.4%0.6500.94078.2%0.6900.6350.5640.564Inertial & MSPC60.1%0.5790.94375.7%0.6430.6090.6010.602Inertial & LMSPC52.5%0.4980.92771.2%0.5180.4820.5250.525 Table 8Average 5-fold results of context dependent placement classification.Test groupSensorACCRECSPEBACCPREMac-F1Mic-F1W-F1PDC-1Inertial98.6%0.8070.95388.0%0.8110.8000.9860.987MFCC98.6%0.7870.96287.4%0.7950.7800.9860.986MSPC97.3%0.5440.89071.7%0.6010.5600.9730.974LMSPC97.4%0.5410.88471.2%0.6680.5750.9740.974Inertial & MFCC98.6%0.8240.96289.3%0.7920.8030.9860.987Inertial & MSPC98.5%0.8090.97089.9%0.8040.8060.9850.986Inertial & LMSPC98.2%0.7550.93784.6%0.7490.7400.9820.982PDC-2Inertial80.5%0.6310.94979.0%0.6700.6420.8050.805MFCC78.1%0.5920.94276.7%0.6200.5990.7810.783MSPC68.3%0.4950.91870.7%0.4960.4880.6830.683LMSPC68.7%0.4850.91870.1%0.4910.4700.6870.687Inertial & MFCC84.7%0.6840.97182.7%0.6770.6700.8470.848Inertial & MSPC82.9%0.6810.95882.0%0.7010.6800.8290.829Inertial & LMSPC83.0%0.6620.95881.0%0.7200.6750.8300.831PDC-3Inertial78.1%0.5910.96978.0%0.5900.5780.7810.782MFCC80.4%0.6290.97280.0%0.5980.5980.8040.805MSPC81.5%0.6160.97379.5%0.6130.6030.8150.815LMSPC79.6%0.6310.97180.1%0.5770.5790.7960.796Inertial & MFCC86.7%0.7000.98184.0%0.7080.6890.8670.867Inertial & MSPC85.1%0.6590.97981.9%0.6560.6460.8510.852Inertial & LMSPC85.0%0.6640.97882.1%0.6880.6670.8500.852PDC-4Inertial51.0%0.4700.92569.7%0.4460.4370.5100.511MFCC50.6%0.5210.92372.2%0.5190.5010.5060.506MSPC39.9%0.3310.90661.9%0.3450.3190.3990.400LMSPC39.0%0.3070.90460.6%0.3390.3060.3900.389Inertial & MFCC56.4%0.6500.94078.2%0.6900.6350.5640.564Inertial & MSPC60.1%0.5790.94375.7%0.6430.6090.6010.602Inertial & LMSPC52.5%0.4980.92771.2%0.5180.4820.5250.525 Average 5-fold results of context dependent placement classification. Discussion The dataset used in our study, popular in various research due to its extensive range of activities that mirror daily life, contains asymmetric real-life data. Consequently, most researchers utilize balanced accuracy for reporting results. Table 9 includes studies employing this real-world dataset, comparing their performance results against our proposed model. Our model evaluation follows two principal approaches. Firstly, user-defined labels are tested at the same hierarchy level without grouping them by primary physical activity. The fusion of audio with inertial sensors demonstrates promising results, reflecting the real-world data structure and imbalanced nature of the dataset. With 5-fold cross-validation, the average balanced accuracy for five-class cases is 90%, surpassing all previously reported results significantly. When the model is extended to include six primary physical activities, with “bicycling” added, it achieves an impressive 91% accuracy in differentiating these activities.Table 9Studies using the real-world ExtraSensory dataset and results.Study#Act.Activity typeSensorsEvaluationClassifierMetricResult925Behavioral contextPA, WA, L,G, PS, AU5-FoldLRBACC72%5051Behavioral contextPA, WA, L,G, PS, AU5-FoldMLPBACC76%515Physical activityPA, WA,G, MHoldoutAdaBoostA88%5253 Physical activityLocation PA, GAU5-FoldCNNF10.520.215422Behavioral contextPA, WA, LG, PS, AUHoldoutLRBACC79%10629Physical activityContext with PPPA, WA, G5-FoldRFBACC83%86%11615Physical activityBehavioral contextPA10-FoldRFBACC80%77%5351329Physical activityBehavioral contextContext with PPPA, WA5-FoldRFBACC80%76%83%Proposed (MGI)61528Physical activityBehavioral contextContext with PPPA, WA,G, M, AU5-FoldCNNBACC91%81%76%Proposed (MGD)51328Physical activityBehavioral contextContext with PPPA, WA,G, M, AU5-FoldCNNBACC90%83%84%*A: accuracy, BACC: balanced accuracy, AU: audio, PS: phone state, G: gyroscope, M: magnetometer, PA: phone accelerometer, WA: watch accelerometer, L: location, MGI: main group independent, MGD: main group dependent, PP: phone placement. Table 9Studies using the real-world ExtraSensory dataset and results.Study#Act.Activity typeSensorsEvaluationClassifierMetricResult925Behavioral contextPA, WA, L,G, PS, AU5-FoldLRBACC72%5051Behavioral contextPA, WA, L,G, PS, AU5-FoldMLPBACC76%515Physical activityPA, WA,G, MHoldoutAdaBoostA88%5253 Physical activityLocation PA, GAU5-FoldCNNF10.520.215422Behavioral contextPA, WA, LG, PS, AUHoldoutLRBACC79%10629Physical activityContext with PPPA, WA, G5-FoldRFBACC83%86%11615Physical activityBehavioral contextPA10-FoldRFBACC80%77%5351329Physical activityBehavioral contextContext with PPPA, WA5-FoldRFBACC80%76%83%Proposed (MGI)61528Physical activityBehavioral contextContext with PPPA, WA,G, M, AU5-FoldCNNBACC91%81%76%Proposed (MGD)51328Physical activityBehavioral contextContext with PPPA, WA,G, M, AU5-FoldCNNBACC90%83%84%*A: accuracy, BACC: balanced accuracy, AU: audio, PS: phone state, G: gyroscope, M: magnetometer, PA: phone accelerometer, WA: watch accelerometer, L: location, MGI: main group independent, MGD: main group dependent, PP: phone placement. Studies using the real-world ExtraSensory dataset and results. PA, WA, L, G, PS, AU PA, WA, L, G, PS, AU PA, WA, G, M 5 3 Physical activity Location PA, G AU 0.52 0.21 PA, WA, L G, PS, AU 6 29 Physical activity Context with PP 83% 86% 6 15 Physical activity Behavioral context 80% 77% 5 13 29 Physical activity Behavioral context Context with PP 80% 76% 83% 6 15 28 Physical activity Behavioral context Context with PP PA, WA, G, M, AU 91% 81% 76% 5 13 28 Physical activity Behavioral context Context with PP PA, WA, G, M, AU 90% 83% 84% *A: accuracy, BACC: balanced accuracy, AU: audio, PS: phone state, G: gyroscope, M: magnetometer, PA: phone accelerometer, WA: watch accelerometer, L: location, MGI: main group independent, MGD: main group dependent, PP: phone placement. At hierarchy level two, multiclass classification is more challenging due to the complexity of activity-context combinations treated as classes. This level features two testing methods: independent and dependent on the main group. The independent method constructs 15 detailed activity classes from various contexts and activities, achieving an 81% balanced accuracy with 5-fold cross-validation, about 4% higher than11, thanks to inertial and audio sensor fusion. In contrast, the dependent method assesses four main groups, each yielding distinct scores. This results in an 83% balanced accuracy, which is 7% better than53. The hierarchical structure of the classifier ends at level three, where smartphone placement plays a critical role in labelling activity-context pairs. Due to the lack of user-labeled data, “running, exercise, phone on pocket” is not included in the pre-processed data. Therefore, level three does not consider this user-defined context. In their study10, researchers utilized three sensor modalities, incorporating multiple features in data processing. They achieved 86% accuracy in level three, main group independent test grouping. While this study’s results at the same configuration did not surpass this score, incorporating audio data enhanced overall system performance when combined with inertial sensors. In main group-dependent test cases, the proposed solution outperforms state-of-the-art solutions by a significant margin. The average score, based on iteration mean results for four activity-dependent groups, reached 85%, almost 2% higher than the reported results in53 (83%). Notably, “sitting” and “standing” scored very high, whereas the remaining two scored extremely low. In contrast, this study’s results for the three groups are very close, with a slightly lower result for “walking”. The proposed solution, designed for real-world HAR, effectively processes data from individuals engaged in routine activities, not predefined scripts. A 20-row window setting enhances activity flow representation while handling the imbalanced dataset. The model concatenates multiple simultaneous activities into a single class for practical application. Inertial sensors in smartphones and smartwatches provide necessary data without extra devices, while audio data, even in smaller segments, is effective alone or in combination with inertial sensors. Both MFCC and MSPC perform well, with varying superiority in certain cases. The study employs a compact deep-learning model, occupying 1.9 to 6.2 MB of memory, dependent on the number of activities and data size, ranging from inertial to audio and fusion images, all with a 20-pixel width and three channels. Table 10 outlines the model’s average memory requirements and input sizes.Table 10Average memory space of model.Input typeHeight (input)Single image size (Input)# Params (model)Memory space (model)Inertial4102 B228,5861.9 MBMFCC14613 B392,4263.2 MBMSPC331.0 KB703,7225.7 MBInertial & MFCC18667 B457,9623.7 MBInertial & MSPC371.1 KB769,2586.2 MB*B: bytes, KB: kilobytes, MB: megabytes, P: pixel. Table 10Average memory space of model.Input typeHeight (input)Single image size (Input)# Params (model)Memory space (model)Inertial4102 B228,5861.9 MBMFCC14613 B392,4263.2 MBMSPC331.0 KB703,7225.7 MBInertial & MFCC18667 B457,9623.7 MBInertial & MSPC371.1 KB769,2586.2 MB*B: bytes, KB: kilobytes, MB: megabytes, P: pixel. Average memory space of model. *B: bytes, KB: kilobytes, MB: megabytes, P: pixel. Following the testing phase, models adapted to various image types have been trained using comprehensive data from 60 users, resulting in a pre-trained model ready for future deployment. Once integrated into a user’s smartphone, this pre-trained lightweight model is poised to facilitate automatic activity recognition. It operates unobtrusively in the background for brief durations, utilising the phone’s embedded sensor data. Table 10 indicates that the model is memory-efficient, consuming only a modest amount of space. Moreover, performance analysis during the testing phase shows that the model’s classification prediction time for a single image is remarkably fast, clocking in at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5 \times 10^{-4}$$\end{document} seconds on an Ubuntu 20.04 virtual machine. In addition, we compared the proposed model with several well-known CNN classifier architectures using the same input as the proposed model. To ensure a fair comparison of inference times, each model was executed 1000 times on an NVIDIA Quadro RTX 5000 GPU, and the average inference time was calculated along with the standard deviation. During this process, PyTorch’s compile functionality was employed to optimize each model for the GPU. The results are presented in Table 11. As demonstrated, the proposed model stands out by achieving an optimal balance between computational efficiency and speed. It achieves a low number of FLOPs (0.0036G) and a significantly faster inference time (0.00048 ms), outperforming all other models in speed, which makes it ideal for real-time applications. Additionally, it has an overparameter count of 0.770M, significantly lower than larger models such as Residual Network (ResNet)50 and ResNet18, which have 23.53M and 11.18M parameters, respectively, along with slower inference times. When compared to popular lightweight models like ShuffleNet and MobileNet, it demonstrates faster inference speeds and competitive FLOPs, with a lower value than MobileNet and slightly higher than ShuffleNet. Overall, the proposed model offers an optimal trade-off between computational complexity and speed, outperforming larger models in efficiency while maintaining competitive performance. This makes it highly suitable for environments where low latency and efficient resource usage are critical.Table 11Comparison of different models in terms of FLOPs, parameters, and inference time.ModelFLOPs (G)Parameters (M)Inference time (ms)Proposed0.00360.7700.00048 ± 0.00003ResNet180.023311.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00194 \pm 0.00007$$\end{document}ResNet500.054223.53\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00437 \pm 0.00037$$\end{document}ShuffleNet v20.00231.27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00490 \pm 0.00014$$\end{document}MobileNet v20.00442.24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00381 \pm 0.00015$$\end{document} Table 11Comparison of different models in terms of FLOPs, parameters, and inference time.ModelFLOPs (G)Parameters (M)Inference time (ms)Proposed0.00360.7700.00048 ± 0.00003ResNet180.023311.18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00194 \pm 0.00007$$\end{document}ResNet500.054223.53\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00437 \pm 0.00037$$\end{document}ShuffleNet v20.00231.27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00490 \pm 0.00014$$\end{document}MobileNet v20.00442.24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.00381 \pm 0.00015$$\end{document} Comparison of different models in terms of FLOPs, parameters, and inference time. Conclusion This study introduces a novel solution for recognizing real-world human activities. Our methodology employs a lightweight model that leverages changes in spatial sensory information to predict user behavior. To address the challenges of multi-label classification, we adopt a hierarchical framework spanning three levels: activity, context, and placement. Additionally, we combat the issue of class imbalance in the dataset by implementing a class-weight approach, emphasizing the significance of thorough data analysis and preprocessing. Our research demonstrates a promising advancement in transforming sensor data into image format, benefiting both inertial and audio sensors. Furthermore, our investigation underscores the enhanced performance achieved by incorporating audio sensors, enabling predictions not only for contexts but also for physical activities and smartphone placement. During the model evaluation phase, we conducted two sets of experiments: one without label grouping and the other involving grouping labels based on their primary physical activity. Although the results at the hierarchical level were similar for both approaches, it is worth noting that, on average, slightly better results were obtained when labels were grouped than the non-grouping cases.