Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition
Haoyu Xie, Haoxuan Li, Chunyuan Zheng, Haonan Yuan, Guorui Liao, Jun Liao, Li Liu · 2025
Read Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition by Haoyu Xie, Haoxuan Li, Chunyuan Zheng, Haonan Yuan, Guorui Liao, Jun Liao, Li Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10917.txt.
Introduction
Seven authors present DecomposeWHAR, a model that reads body-worn sensors to name the activity a person is performing. Using several sensors at once beats one, but most models run the same filters over every sensor channel and blur the links within and between sensors. The new model splits the job into two phases. A decomposition phase builds a rich profile of each channel on its own, keeping what makes it distinct.
The fusion phase then works in stages. It links channels inside each sensor, blends them at two levels, and uses a state space model to catch long-range timing patterns. A self-attention step finally reads the ties across sensors. On three standard activity datasets, the model beat the best prior systems while keeping its computing cost acceptable.
Catalog record
| CREATOR | Haoyu Xie, Haoxuan Li, Chunyuan Zheng, Haonan Yuan, Guorui Liao, Jun Liao, Li Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10917
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition Haoyu Xie, Haoxuan Li, Chunyuan Zheng, Haonan Yuan, Guorui Liao, Jun Liao, Li Liu Abstract Wearable Human Activity Recognition (WHAR) is a prominent research area within ubiquitous computing. Multi-sensor synchronous measurement has proven to be more effective for WHAR than using a single sensor. However, existing WHAR methods use shared convolutional kernels for indiscriminate temporal feature extraction across each sensor variable, which fails to effectively capture spatio-temporal relationships of intra-sensor and inter-sensor variables. We propose the DecomposeWHAR model consisting of a decomposition phase and a fusion phase to better model the relationships between modality variables. The decomposition creates high-dimensional representations of each intra-sensor variable through the improved Depth Separable Convolution to capture local temporal features while preserving their unique characteristics. The fusion phase begins by capturing relationships between intra-sensor variables and fusing their features at both the channel and variable levels. Long-range temporal dependencies are modeled using the State Space Model (SSM), and later cross-sensor interactions are dynamically captured through a self-attention mechanism, highlighting inter-sensor spatial correlations. Our model demonstrates superior performance on three widely used WHAR datasets, significantly outperforming state-of-the-art models while maintaining acceptable computational efficiency. 1 School of Big Data and Software Engineering, Chongqing University, China 2 Center for Data Science, Peking University, China 3 School of Mathematical Sciences, Peking University, China 4 School of Computer Science and Engineering, Beihang University, China {haoyuxie,guoruiliao}@stu.cqu.edu.cn, {hxli,cyzheng}@stu.pku.edu.cn, [email protected], {liaojun,dcsliuli}@cqu.edu.cn Introduction Figure 1: Intra- and Inter-Sensor Variables in WHAR. Human Activity Recognition (HAR) methods can be categorized into vision-based, environment-based, and wearable sensor-based approaches. Vision-based HAR leverages video data but is restricted by lighting conditions and camera coverage. Environment-based HAR uses ambient sensors such as sound or WiFi, though it faces limitations in location and often struggles with accuracy. Wearable Human Activity Recognition (WHAR), however, involves attaching sensors directly to the body. This approach offers continuous, direct measurement of motion, is less affected by environmental factors, and is versatile across different settings. It benefits from the use of readily available devices like smartphones and watches, making it practical for applications in healthcare and sports. WHAR can be classified as a multivariate time series classification problem. The multivariate aspect arises from the various modalities and axes of sensor variables. A typical motion sensor might include a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer, resulting in a 9-variable time series data sample. In WHAR, there are two types of variable relationships: Intra- and Inter-Sensor Modality Variables in Figure 1 . Intra-sensor variables refer to different variables within the same sensor, while inter-sensor variables refer to variables from various sensors located at different positions on the human body. Failing to disentangle these relationships while extracting features will lead to sub-optimal performance. The first key challenge in WHAR is to capture local and global temporal features of Intra-Sensor Modality Variables while handling the intricate relationships between these diverse variables. Previous studies mainly include two types of approaches. The first type of approach, exemplified by DeepConvLSTM ( Ordóñez and Roggen 2016 ) and Attend and Discriminate ( Abedin et al. 2021 ) , utilizes shared convolutional kernels to capture local temporal patterns within each variable, followed by Long Short-Term Memory Networks (LSTMs) to integrate variables and model global temporal dependencies. The second type of approach ( Miao et al. 2022 ; Ahmad and Leung 2024 ) employs Conv1D multimodal fusion before temporal extraction, which can lead to the loss of temporal information within each variable. These methods usually pay less attention to retaining the high-level features intrinsic to each variable and modeling complex cross-variable interactions, potentially limiting their ability to fully exploit rich multi-modal intra-sensor data and resulting in reduced recognition performance. The second challenge is to capture the Inter-Sensor spatio-temporal correlations. For instance, during running, there is a strong correlation between the forward swing of one side’s hand and the forward stride of the opposite leg. Similarly, recognizing the relationship between stationary hands and moving legs during cycling can greatly improve activity recognition performance ( Ahmad and Leung 2024 ) . However, the methods mentioned above typically use dense layers or Recurrent Neural Network (RNN) layers to indiscriminately fuse all variables from all sensors. This simplistic fusion approach fails to capture the spatial relationships between sensors and lacks interpretability. To address this, DynamicWHAR ( Miao et al. 2022 ) separates sensors placed at different positions and performs multimodal fusion and temporal feature extraction within each sensor individually. It then utilizes a Graph Convolutional Network (GCN) to capture dynamic correlations between sensors. However, GCNs face limitations due to their reliance on predefined graph structures, which restrict their ability to model dynamic relationships and scale effectively with large graphs ( Yuan et al. 2024 ) . Another limitation is that GCNs compute correlations between sensors symmetrically, which can overlook the directional and asymmetric nature of interactions between sensors. Additionally, GCNs are less efficient in parallelizing computations due to their message-passing mechanisms ( Yuan et al. 2023 ) . To overcome these limitations, we introduce a novel framework consisting of a Modality-Aware Signal Decomposition phase and a Hierarchical Interaction Fusion phase. For the first challenge, Modality-Aware Signal Decomposition first separates different sensors to focus on the intra-sensor modality interactions. Inspired by a modern pure convolution structure for time series ( Donghao and Xue 2023 ) , we leverage a modality-specific embedding layer and local temporal extraction module through Depth-Wise Convolution to independently extract implicit high-level temporal features from intra-sensor modality variable. This ensures both sensor level and variable level independence, capturing detailed temporal features of each modality. In the Hierarchical Interaction Fusion phase, we use Point-Wise Convolution to integrate the decomposed channels and modality variables, effectively capturing the intricate dependencies among these intra-sensor modality variables. To address the second challenge, we introduce Global Temporal Aggregation and Cross-Sensor Interaction, implemented by the Mamba-attention module, our model consolidates features across the entire temporal dimension and dynamically captures inter-sensor spatio-temporal correlations. Our main contributions are as follows: • We propose a novel framework that incorporates sensor level, temporal level, variable level, and channel level decomposition, followed by reverse-order hierarchical fusion. Our approach effectively captures spatio-temporal relationships between intra- and inter-sensor modality variables, enhancing the recognition of complex patterns in multi-sensor data and diverse scenarios. • To preserve the unique temporal features of intra-sensor modalities, we enhance Depth Separable Convolution to independently extract high-level temporal representations from each modality, mitigating interference between intra-sensor modalities. • We introduce a learnable Mamba-attention module to model inter-sensor correlations dynamically. This module adjusts the relative importance of sensors, addressing heterogeneity across different activities and subjects and enabling more accurate recognition. • Our model achieves superior recognition performance while maintaining computational efficiency, significantly surpassing state-of-the-art baselines. Related Works Temporal Feature Extraction for WHAR Effective temporal feature extraction is essential for WHAR, capturing dynamic patterns from sequential sensor data. Early WHAR methods employed traditional machine learning techniques such as decision trees, SVMs, and k-NN ( Attal et al. 2015 ) . These approaches, limited by feature engineering and temporal modeling, led to the adoption of deep learning methods ( Li et al. 2021a ) . Deep learning has significantly enhanced HAR through Convolutional