GraMFedDHAR: Graph Based Multimodal Differentially Private Federated HAR
Labani Halder, Tanmay Sen, Sarbani Palit · 2025
Read GraMFedDHAR: Graph Based Multimodal Differentially Private Federated HAR by Labani Halder, Tanmay Sen, Sarbani Palit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05671.txt.
Introduction
Recognising human activity from several body-worn and ambient sensors, without centralising private data, is the aim of Labani Halder, Tanmay Sen, and Sarbani Palit in a 2025 paper. Labels are scarce, sensors are noisy or incomplete, and pooling raw data raises privacy problems.
Their framework models each sensor stream, from pressure mats to depth cameras and accelerometers, as its own graph. Residual graph networks process the streams, and an attention step fuses them by importance rather than simple joining. Training is federated with differential privacy, so only protected updates are shared. The graph model beat a feed-forward baseline modestly without privacy noise and by wide margins under privacy budgets.
Catalog record
| CREATOR | Labani Halder, Tanmay Sen, Sarbani Palit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GraMFedDHAR: Graph Based Multimodal Differentially Private Federated HAR Labani Halder, Tanmay Sen, Sarbani Palit Abstract Human Activity Recognition (HAR) using multimodal sensor data remains challenging due to noisy or incomplete measurements, scarcity of labeled examples, and privacy concerns. Traditional centralized deep learning approaches are often constrained by infrastructure availability, network latency, and data sharing restrictions. While federated learning (FL) addresses privacy by training models locally and sharing only model parameters, it still has to tackle issues arising from the use of heterogeneous multimodal data and differential privacy requirements. In this article, a Graph-based Multimodal Federated Learning framework, GraMFedDHAR, is proposed for HAR tasks. Diverse sensor streams such as a pressure mat, depth camera, and multiple accelerometers are modeled as modality-specific graphs, processed through residual Graph Convolutional Neural Networks (GCNs), and fused via attention-based weighting rather than simple concatenation. The fused embeddings enable robust activity classification, while differential privacy safeguards data during federated aggregation. Experimental results show that the proposed MultiModalGCN model outperforms the baseline MultiModalFFN, with up to 2% higher accuracy in non-DP settings in both centralized and federated paradigms. More importantly, significant improvements are observed under differential privacy constraints: MultiModalGCN consistently surpasses MultiModalFFN, with performance gaps ranging from 7–13% depending on the privacy budget and setting. These results highlight the robustness of graph-based modeling in multimodal learning, where GNNs prove more resilient to the performance degradation introduced by DP noise. 1 Introduction Human Activity Recognition (HAR) is now central to applications in smart homes, fitness tracking, healthcare, and human-computer interaction. However, real-world HAR systems face few persistent challenges: heterogeneous data, privacy preservation, and robust multimodal learning. Practical HAR systems must deal with diverse sensors, missing modalities, and client-specific constraints [ 1 , 2 , 3 , 4 ] . Centralized ensemble and fusion methods are effective in controlled environments but are often impractical for federated learning. This is because modalities are distributed across clients and cannot be centrally concatenated for training [ 1 ] . A robust framework is required to be developed that can combine different data types and work even when some data is missing. Another major challenge is privacy preservation for multi-modal sequential data [ 5 , 6 , 7 ] . Although federated learning (FL) mitigates privacy risks by training models locally and sharing only model updates, it is still vulnerable to inference attacks and leakage of sensitive activity patterns. Traditional Differential Privacy (DP), designed for i.i.d. data, struggles to account for temporal correlations, often misestimating privacy loss in continuous activity streams [ 8 , 6 , 5 ] . Thus, HAR requires federated solutions that are not only multimodal-aware but also enhanced with temporally sensitive privacy guarantees. Multimodal HAR benefits from graph-based modeling, which naturally captures spatial and relational dependencies across different sensor modalities. Graph Convolutional Networks (GCNs) have shown strong potential in capturing contextual structure in HAR tasks [ 9 , 10 , 11 , 12 , 13 ] . However, adapting GCNs to FL settings introduces new risks, as sharing graph parameters can leak structural information [ 14 , 13 ] . Developing communication-efficient and privacy-aware federated GCN protocols is therefore essential. Existing HAR datasets and benchmarks lack realism because they often fail to include key real-world factors such as non-IID data distributions, systematic data loss from certain sensors, unsynchronized data collection, and device-level constraints like limited energy and latency [ 15 , 16 , 10 , 17 ] . HAR systems also face additional concerns including high labeling costs, vulnerability to adversarial attacks, and the need for explainability and fairness in sensitive domains such as healthcare [ 13 , 7 , 11 , 4 ] . These broader challenges highlight the importance of developing multimodal HAR frameworks that preserve privacy, are robust, and are trustworthy. In this article, we propose GraMFedDHAR, a Graph-based Multimodal Federated Learning framework for differentially private HAR. In our framework, each sensor stream (e.g., pressure mats, depth cameras, accelerometers) is represented as a modality-specific graph and processed through residual GCNs. Their embeddings are fused via attention-based weighting, enabling robust multimodal activity classification. To protect sensitive information, DP is integrated into the FL aggregation process, ensuring user-level protection. Extensive experiments demonstrate that graph-based multimodal modeling is significantly more resilient than feedforward baselines under DP constraints. The major contributions of this work are summarized as follows: • A novel hybrid graph construction strategy for multimodal HAR is proposed. This approach models temporal dependencies by linking consecutive time windows, while spatial dependencies across sensor modalities are captured through a distance-based adjacency formulation. • DP is integrated into the FL process. This is achieved by perturbing client-side updates before aggregation, which provides formal user-level privacy guarantees while preserving recognition performance. • It may also be emphasized that this work is the first to perform the integration of DP and FL on the multimodal graph based HAR task. 2 Related Works In the existing literature, the problem of HAR is typically categorized as a traditional machine learning and deep learning based supervised or unsupervised classification problem. Traditional machine learning approaches using smart sensors and wearable devices for healthcare monitoring, leverage body motion and vital sign data to recognize daily activities. However, performance is often limited by the variability of the data set and the lack of a universally optimal algorithm [ 18 ] . A comprehensive review on HAR is provided in [ 15 ] . Apart from highlighting the challenges of accurately detecting complex activities, it further analyzes vision-based and non-vision-based sensors, along with various traditional machine learning techniques applied to different HAR applications. [ 19 ] proposes Deep-ConvLSTM that learns time-dependent sequence context using a self-attention mechanism for the HAR problem, with applications in healthcare. Recently, Transformer-based models [ 20 ] have gained significant traction for addressing time series problems like HAR. [ 16 ] proposed the Human Activity Recognition Transformer (HART), a lightweight, sensor-wise Transformer optimized for IMU-based HAR on mobile devices, achieving higher accuracy than state-of-the-art methods. Mondal et al. [ 9 ] proposed a transductive graph-based framework for the HAR task by converting multi-variate time series sensor data into a graph. The model’s robustness is also verified on imbalanced data. A comprehensive survey on skeleton-based action recognition using graph convolutional neural networks for video data is provided in [ 14 ] . [ 11 ] introduces body RFID skeleton-based activity recognition using GCN, where activities are modeled as RFID skeleton activity graphs achieving superior recognition performance compared to existing HAR methods. [ 12 ] proposes a deep CNN to automatically extract discriminative features from optimal activity graphs for HAR, effectively capturing correlations between activities. Multi-modal activity recognition has gained significant attention for its ability to leverage complementary information and improve robustness to missing or corrupted data. [ 1 ] proposes MMHAR-EnsemNet, a multi-modal HAR framework that integrates skeleton, RGB, and inertial signals using CNN and LSTM-based networks, with outputs combined in an ensemble achieving superior performance for experiments on UTD-MHAD and Berkeley-MHAD datasets. The authors in [ 2 ] propose a unified FL framework with co-attention–based fusion and Model-Agnostic Meta-Learning (MAML) driven personalization, demonstrating superior performance on multimodal activity recognition tasks. A federated learning (FL) framework MHARFedLLM for multi-modal HAR using a federated large language model by