Recognition of Daily Activities Through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living Authors: Kooshan Hashemifard, Pau Climent-Pérez, Francisco Florez-Revuelta Published: 2026 ABSTRACT Recognition of daily activities is a critical element for effective Ambient Assisted Living (AAL) systems, particularly to monitor the well-being and support the independence of older adults in indoor environments. However, developing robust activity recognition systems faces significant challenges, including intra-class variability, inter-class similarity, environmental variability, camera perspectives, and scene complexity. This paper presents a multi-modal approach for the recognition of activities of daily living tailored for older adults within AAL settings. The proposed system integrates visual information processed by a 3D Convolutional Neural Network (CNN) with 3D human pose data analyzed by a Graph Convolutional Network. Contextual information, derived from an object detection module, is fused with the 3D CNN features using a cross-attention mechanism to enhance recognition accuracy. This method is evaluated using the Toyota SmartHome dataset, which consists of real-world indoor activities. The results indicate that the proposed system achieves competitive classification accuracy for a range of daily activities, highlighting its potential as an essential component for advanced AAL monitoring solutions. This advancement supports the broader goal of developing intelligent systems that promote safety and autonomy among older adults. Introduction The demographic shift towards an aging global population presents an increasing need for technologies that support older adults to live independently and safely within their own homes [1]. Ambient Assisted Living (AAL) systems aim to address this challenge by providing solutions that monitor well-being, improve safety, and offer timely assistance when needed. Automatic recognition of daily activities is therefore crucial for developing intelligent monitoring systems that can adapt their behavior based on situational needs, providing detailed information when necessary for safety while maintaining privacy during routine daily activities. In this domain, indoor Human Activity Recognition (HAR) specifically addresses activities carried out in confined environments, making it particularly relevant for AAL applications that involve continuous monitoring. This work focuses on the recognition of activities of daily living (ADLs) for older adults, which varies across diverse indoor environments such as homes, hospitals, care facilities, and assisted living centers. These activities range from basic ones like walking, sitting, sleeping, and drinking water, to complex multi-step sequences such as “taking medication” which can involve a series of simpler activities like “opening the pill box” and “drinking water,” or “preparing meals” which may include food handling activities such as cooking and stirring [2]. Our approach contributes to advancing such recognition capabilities specifically for context-aware privacy-preserving monitoring systems in residential settings. Indoor activities present several key challenges: intra-class variability, which occurs when different individuals perform the same activity in distinct ways (for instance, “drinking water” while sitting, standing, or walking may generate different recognition patterns), inter-class similarity where distinct activities share comparable motion patterns (such as stirring tea versus stirring soup, or jumping and dancing which may have similar movements that are ambiguous for recognition systems), view variance which significantly impacts recognition performance when cameras are positioned at different angles or heights, and object interaction complexity where many daily living activities are inherently defined by how humans interact with household objects. In addition, environmental factors such as varying lighting conditions, occlusions, camera perspective, and scene complexity further complicate robust recognition in real-world indoor settings. The unique nature of indoor activities means that conventional methods designed for general HAR may not be suitable, as the activity set differs from one environment to another, and one model cannot fit all scenarios. This specificity makes indoor HAR a distinct research track that requires activity recognition problems to be considered differently based on the application area, particularly for concepts such as smart homes, automated patient care, and older adults care systems. Advances in computer vision and deep learning have provided powerful tools for HAR. 3D Convolutional Neural Networks (CNNs) have emerged as effective feature extraction backbones due to their ability to capture both spatial and temporal patterns in video data through hierarchical feature learning [3]. However, traditional 3D CNNs with uniform spatio-temporal kernels applied across entire video volumes often lack the flexibility needed to address the nuanced challenges of indoor activities, particularly struggling with view variance where similar actions performed from different angles or with subtle variations are not well distinguished. To counteract this problem, human pose data has been used to provide view-invariant geometric information that remains consistent across different camera perspectives, making it valuable for robust activity recognition [4, 5]. Graph Convolutional Networks (GCNs) for analyzing skeletal pose information have shown promise in capturing spatio-temporal dynamics, particularly for addressing view variance challenges [6, 7]. However, pose-based methods often struggle to differentiate activities with similar poses but different contexts, as they fail to effectively leverage complementary visual features and object interactions present in video data, limiting their effectiveness for fine-grained activity discrimination. Multi-modal systems that integrate information from various sources such as video, pose, and object interactions are increasingly being explored to enhance contextual understanding and improve the disambiguation of activities [8]. The integration of object context is particularly relevant for indoor daily activities, where the manipulation of objects often defines the action itself [9], although it has received comparatively less attention in previous research. This paper presents a multi-modal daily activity recognition system designed for older adults in indoor AAL environments. The system uses a 3D CNN to extract spatiotemporal features from video streams, a GCN to model dynamic human pose from 3D skeleton data, and an object detection module to capture information about human–object interactions. These complementary data streams are then integrated through a cross-attention mechanism, allowing the model to prioritize the most relevant features for activity classification. The proposed framework is evaluated on the Toyota SmartHome dataset [10], which includes unscripted daily activities performed by older adults in a homelike setting and recorded with multiple cameras. This makes the dataset particularly suitable for assessing activity recognition methods intended for AAL applications. The contributions of this work are threefold, each addressing specific challenges in indoor AAL activity recognition: First, we propose a multi-modal architecture that integrates visual information (3D CNN), human pose data (GCN), and scene context (object detection) through a cross-attention mechanism. This directly addresses the challenge that many indoor activities are fundamentally defined by object interactions, as our system can distinguish between similar motions based on the objects involved, improving recognition of daily living activities where context matters. Second, we utilize a spatial embedding approach that establishes correspondences between 3D pose data and visual features. This tackles the view variance problem common in indoor monitoring by aligning pose information (which is naturally view-invariant) with visual features, enabling our system to maintain recognition accuracy across different camera positions and angles typical in home environments. Third, our 3D CNN backbone with cross-attention enhancement learns both local motion patterns and global activity structure while being guided by object context. This addresses the limitation of traditional CNNs that apply uniform processing across video volumes. Our approach adapts feature processing based on relevant objects and pose information, leading to more discriminative representations for complex indoor activities. Consequently, this work advances accurate, context-aware activity recognition for AAL, a prerequisite for effective and privacy-sensitive monitoring that supports older adults in daily life. Continuous observation can threaten dignity and autonomy, so systems must adapt what they sense and store to the situation. Context-aware methods (e.g., [11]) estimate salient variables such as appearance (e.g., degree of nudity), relevant events (e.g., a fall), and ongoing activities. HAR plays a central role in this process: by inferring what a person is doing, the system can adjust the intensity of monitoring and the granularity of data retained. Our approach follows this principle, providing detailed analysis only when safety demands it and defaulting to lightweight, low-visibility operation during routine activities. In doing so, it balances safety with privacy, offering actionable assistance while minimising unnecessary data exposure. The remainder of this paper is organized as follows. Section 2 will discuss related work in human activity recognition, particularly focusing on multi-modal approaches and systems applicable to AAL scenarios. Section 3 will detail the architecture of the proposed activity recognition system. Section 4 will describe the experimental setup, the dataset used, and the evaluation metrics. Section 5 presents and discusses the results, and finally, Section 6 concludes the article and suggests directions for future research. Related Work Indoor HAR has advanced along four converging research threads: (i) strengthening spatio-temporal visual modeling beyond uniform 3D convolutions, (ii) leveraging skeleton streams to counter view variance while exposing motion geometry, (iii) incorporating object and scene context to disambiguate fine-grained ADLs, and (iv) designing multi-modal fusion and cross-modal alignment mechanisms that go beyond late concatenation toward learned correspondence and attention. These lines are particularly important for ADL recognition, where inter-class similarity and intra-class variability meet strong object dependence and viewpoint changes [2, 12]. Visual Backbone Evolution: From 3D CNNs to Transformers Classical 3D CNNs and multi-stream CNNs established strong baselines for video recognition, but their uniform spatio-temporal processing often blurs subtle discriminative cues and struggles with view shifts in cluttered indoor scenes [13–15]. Inflated 3D convolutions improved spatio-temporal modeling, yet still inherit sensitivity to camera geometry and weak grounding in object-centric semantics [3]. Non-local/self-attention operators add long-range dependencies but may diffuse focus in complex rooms when actor-relevant tokens are not explicitly guided [16, 17]. Recent video transformers such as ViViT address global context with space-time attention [18, 19], but aside from the need for large datasets, uni-modal RGB remains prone to scene bias and viewpoint sensitivity in ADL recognition. Alternative attention mechanisms have been explored, with Glimpse Clouds [20] using unstructured attention to extract visual glimpse sequences without spatial coherence constraints, leveraging pose supervision during training to focus attention on human anatomical structures. Skeleton-Based Recognition: From Handcrafted Features to Graph Networks Skeleton-based human action recognition has evolved from handcrafted features to deep learning approaches. The early methods relied on depth sensors and spatiotemporal features for activity recognition [21, 22]. The introduction of Graph Convolutional Networks (GCNs) marked a significant advancement, with ST-GCN [7] first modeling skeleton data as spatiotemporal graphs with joints as vertices and natural body connections as edges. Building upon ST-GCN, several improvements have been proposed. Directed Graph Neural Networks [6] introduced directed acyclic graphs to better model kinematic dependencies between joints and bones. Two-Stream Adaptive Graph Convolutional Networks [23] combined first-order (joint coordinates) and second-order (bone vectors) information while learning adaptive graph topologies. Recent work has extended skeleton graphs with non-adjacent joint connections and improved partitioning strategies [24], while multi-scale approaches have been explored [25]. Multi-stream architectures have also gained attention, with three-stream networks combining spatial-temporal graphs, tree-structure-reference-joints images, and RGB appearance data [26]. Graph self-attention networks like MGSAN [27] demonstrate that combining GCNs with self-attention can capture intrinsic topology and long-term temporal dependencies between skeleton joints, moving beyond traditional 2D spatial plus 1D temporal approaches to true spatiotemporal modeling. Cross-Modal Training Strategies: Skeleton-Informed RGB Learning A central insight for indoor ADL recognition is that skeletal information should shape the visual representation space during training, even when deployment relies solely on RGB input. Several approaches leverage pose information during training while maintaining RGB-only inference. Pose-induced transformers [28] explicitly inject 2D/3D skeleton priors into video transformers through auxiliary token–joint mapping and feature alignment, then remove pose modules at inference to retain efficiency. Built upon the TimeSformer backbone [18], π -ViT introduces plug-in modules (2D-SIM and 3D-SIM) that perform pose-aware auxiliary tasks during training, enabling the video transformer to learn pose-augmented RGB representations without requiring poses during deployment. The approach implements cross-modal alignment as two auxiliary tasks: a token–joint mapping that teaches which visual tokens correspond to joints, and a skeleton feature alignment/classification that enforces RGB tokens to carry view-agnostic motion structure. Complementary to this approach, self-supervised masked pretraining guided by skeleton latents likewise sculpts RGB representations to internalize motion regularities, enabling RGB-only fine-tuning and inference. For instance, SV-data2vec [29] employs a teacher-student architecture where the teacher generates contextualized targets from skeleton data while the student performs masked prediction on both skeleton and visual inputs, ultimately allowing fine-tuning with RGB data alone. After pretraining, the model fine-tunes with RGB-only, matching or surpassing RGB+pose methods on several protocols. Multi-Modal Fusion and Attention Mechanisms Score averaging and feature concatenation are common fusion strategies, but they often fall short when RGB, pose, and object streams are misaligned in space, time, or geometry [12, 30]. To tackle this problem, VPN and VPN++ [8, 31] introduced enhanced video–pose integration modules that improve performance in indoor activity recognition. Attention-based multi-stream architectures explicitly route information through learned connections to balance complementary modalities. Cross-modal transformers like STAR-transformer [32] demonstrate the effectiveness of jointly processing video and skeleton features through specialized spatio-temporal cross-attention mechanisms, aggregating global grid tokens from video frames and joint map tokens from skeleton sequences into unified multi-class representations. The Convolutional Block Attention Module (CBAM) has emerged as a particularly effective hybrid attention mechanism, combining channel and spatial attention to enhance feature extraction in multi-modal frameworks. CBAM-based approaches like CBAM-MFFAR [33] achieve state-of-the-art performance by applying attention at multiple fusion stages, including early fusion through bidirectional lateral connections and late fusion of prediction scores. Model-based fusion strategies have proven superior to simple concatenation or late fusion approaches. MMNet [34] demonstrates that transferring learned attention weights from skeleton modalities to guide RGB feature extraction significantly outperforms generic distillation methods, confirming that explicit correspondence between modalities matters more than simple logits transfer. Fusion-GCN extends this principle by directly incorporating additional sensor modalities into skeleton graph structures through either channel dimension fusion or spatial dimension fusion, achieving substantial improvements on large-scale multi-modal datasets [35]. Wearable sensor-based systems further demonstrate attention’s versatility across modalities. Deep CNN-LSTM architectures with self-attention mechanisms achieve remarkable performance on smartphone sensor data, with models by learning temporal correlations between sensor readings [36]. TCN-attention frameworks leverage temporal convolutional networks with attention mechanisms to capture long-range dependencies in sensor time series, significantly outperforming traditional CNN-based approaches on multiple wearable sensor benchmarks [37]. Contemporary 3D CNN approaches like spatiotemporal multi-modal learning frameworks demonstrate that attention-guided depth and pose representations can be effectively combined through two-stream architectures, where each modality provides complementary spatiotemporal information [38]. Object and Scene Context Integration Many ADL categories are fundamentally object-defined, so integrating detectors and modeling human–object relations is crucial [2]. Region-based and one-stage detectors provide reliable cues that can be used to guide attention and token selection in video backbones [39, 40]. This has been widely done in egocentric ADL recognition [41]; however, it is not as common in third-person vision-based HAR. AssembleNet++ [9] introduces peer-attention mechanisms that wire attention connections across modalities, with particular emphasis on object relations through omnipresent connectivity from object input blocks. Recent systems leverage pose-aware saliency to focus on hands and manipulated items; π -ViT’s 2D token–joint supervision naturally complements object priors by tethering attention to anatomy-grounded regions. Context-based representations can dominate over motion cues in action recognition, leading to scenarios where visually similar actions are distinguished primarily through background objects rather than human motion patterns [42]. These training-time aligners are particularly effective against intra-class variability and inter-class similarity endemic to indoor ADL recognition [2]. Proposed Approach Based on the literature review, indoor ADL recognition benefits from: (i) strong spatio-temporal vision backbones, (ii) skeleton guidance (anatomy-aware) to stabilize against view variance and occlusions, (iii) object priors for human-object interactions, and (iv) cross-attention mechanisms that provide additional contextual information to mitigate intra-class variability and inter-class similarity. For elderly-focused indoor HAR, while transformers demonstrate strong performance in ADL recognition, limited training data constrains their effectiveness. This motivates adopting cross-attention and correspondence mechanisms from transformers within more data-efficient architectures, leveraging their strengths while avoiding their data requirements. Building on these insights, we propose a multi-modal activity recognition framework that integrates RGB video processed through a 3D CNN, 3D human pose data analyzed via a GCN, and object context information through pre-trained detection. These modalities are fused using cross-attention with explicit spatial embedding between pose and visual features to enable robust activity classification suitable for data-constrained elderly ADL scenarios. The overall architecture is illustrated in Fig. 1 and consists of four main components: (1) data preprocessing for video and pose streams, (2) feature extraction networks for each modality, (3) a cross-attention fusion module, and (4) a classification head for final activity prediction. Details for each component are provided below.Fig. 1Architecture of the proposed method Architecture of the proposed method Video and Pose Data Preprocessing To address the challenges of view variance and environmental variability in indoor activity recognition, preprocessing techniques are applied to normalise both skeletal and video data streams. These steps, inspired by [43], aim to reduce intra-class variability while preserving the essential characteristics required for robust activity classification. The 3D human pose data undergoes a two-stage rotation process to create view-invariant skeletal representations (Fig. 2). This normalization is crucial for indoor HAR systems where the same activity may be captured from different camera perspectives, leading to unnecessary variations in the skeletal joint positions. First, a Y-axis rotation is applied to ensure all skeletons are oriented to “face forward” regardless of the original camera viewpoint. The rotation angle α is calculated using three key joints - left shoulder (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {s_l}$$\end{document}), right shoulder (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {s_r}$$\end{document}), and right hip (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {h_r}$$\end{document}) - which define the torso plane:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \alpha = \arctan \left( \frac{s_{lz} - \frac{s_{rz} + h_{rz}}{2}}{s_{lx} - \frac{s_{rx} + h_{rx}}{2}}\right) \end{aligned}$$\end{document}This rotation is computed once at the beginning of each sequence (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_0$$\end{document}) to maintain natural body rotations that occur as part of the activity itself. Second, a Z-axis rotation compensates for camera tilt with respect to the ground plane. This rotation β is calculated at each frame to account for temporal variations in pose estimation:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \beta _t = \frac{\beta _{s,t} + \beta _{h,t}}{2} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{s,t}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{h,t}$$\end{document} represent independent angle estimations from shoulder and hip joint pairs respectively. This frame-by-frame correction ensures consistent skeletal orientation throughout the activity sequence.Fig. 2Sequential skeleton rotation stages adapted from [43]. (a) Original frame; (b) left: base pose, center: Y-axis rotated pose to face forward, right: fully rotated pose (Y and Z axes) Sequential skeleton rotation stages adapted from [43]. (a) Original frame; (b) left: base pose, center: Y-axis rotated pose to face forward, right: fully rotated pose (Y and Z axes) Complementing the skeletal normalization, the RGB video preprocessing employs a “full activity crop” strategy that captures the complete spatial context of human activities. Compared to only cropping around an individual person, this method considers the entire spatial footprint of the activity over time. The full activity crop is defined by identifying the minimum and maximum coordinates of all human detections throughout the video sequence:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \vec {p_{TL}} = \min _{i=1..t}(x_i, y_i), \quad \vec {p_{BR}} = \max _{i=1..t}(x_i, y_i) \end{aligned}$$\end{document}This creates a bounding box from the top-left (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {p_{TL}}$$\end{document}) to bottom-right (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {p_{BR}}$$\end{document}) corners, that encompasses the complete activity space, which is particularly beneficial for activities where spatial displacement is a key discriminative feature (e.g., walking, moving between locations). Since the neural network architecture requires a square input, a square crop is generated by finding the center of the activity bounding box and using the larger dimension as the square side length. When the resulting crop extends beyond the image boundaries, padding with neutral gray values (RGB: 128) maintains the aspect ratio while preserving the spatial relationships within the activity region. This preprocessing approach addresses two key limitations: it provides more accurate person detection through improved bounding box estimation, and it preserves the spatial dynamics of activities that may be lost when cropping around individual frame detections. The combination of view-invariant skeletal normalization and comprehensive spatial context preservation in video data creates a more robust foundation for multi-modal activity recognition in indoor environments. Feature Learning Components Following the preprocessing stage, each modality undergoes specialized feature extraction to capture its unique characteristics. The visual stream employs a 3D Convolutional Neural Network (I3D) [3] to extract spatio-temporal features from the normalized video data, while the pose stream utilizes GCN to model pose information and temporal dynamics in the skeletal data. The object detection component provides semantic context by identifying relevant objects within the activity region. Additionally, a pre-trained object detection module identifies and localizes relevant objects within the activity region to provide contextual information for attention-based fusion. Visual Feature Extraction with I3D The RGB video component of our multi-modal framework employs I3D as the feature extraction backbone for processing the preprocessed RGB video data. Following the preprocessing stage, the input consists of temporally-ordered human activity crops with dimensions corresponding to the full activity spatial extent, determined during the preprocessing phase. The I3D network processes sequences of T sequential frames, sampled from the preprocessed video V, where each frame contains the square-cropped activity region. The spatio-temporal representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f}$$\end{document} is extracted from an intermediate layer of the I3D network, specifically the convolutional layer preceding the Global Average Pooling (GAP) operation. This choice preserves both spatial and temporal granularity necessary for subsequent attention-based fusion with pose and object context information. The resulting feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f}$$\end{document} has \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t \times h \times w \times c$$\end{document} dimensions, where t represents the temporal dimension, h × w corresponds to the spatial resolution of the feature map, and c denotes the number of feature channels. This 4-dimensional representation captures hierarchical spatio-temporal patterns that encode both local motion dynamics and global activity structure, making it suitable for distinguishing between activities with similar motion patterns but different spatial contexts [3]. The I3D backbone uses pre-trained weights from large-scale action recognition datasets, providing robust feature representations that transfer effectively to the indoor AAL domain. The extracted features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f}$$\end{document} serve as input to the cross-attention mechanism, where they are spatially and temporally modulated based on complementary information from the pose and object detection modalities. Pose Feature Extraction with GCN To address the view variance challenges inherent in indoor AAL environments, our framework incorporates a specialized pose analysis component that leverages the natural structural properties of human skeletal data. The skeletal modality provides complementary geometric information that remains consistent across different camera viewpoints, making it particularly valuable for robust activity recognition in home monitoring scenarios. Human pose sequences are represented as a temporal collection of 3D joint coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P \in \mathbb {R}^{3 \times J \times T}$$\end{document}, where each frame \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_t \in \mathbb {R}^{3 \times J}$$\end{document} contains the spatial positions of J anatomical joints. Rather than treating these coordinates as independent features, we exploit the inherent connectivity structure of the human body by modeling each pose frame as an undirected graph where joints serve as vertices and anatomical connections define the edge relationships. The graph representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_t(P_t, A)$$\end{document} at each temporal step t incorporates a weighted adjacency matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A \in \mathbb {R}^{J \times J}$$\end{document} that encodes the natural biomechanical relationships between body parts:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{ij} = {\left\{ \begin{array}{ll} 0, & \text {if } i = j \\ \alpha , & \text {if joint } i \text { and joint } j \text { are connected} \\ \beta , & \text {if joint } i \text { and joint } j \text { are disconnected}\\ \end{array}\right. } \end{aligned}$$\end{document}This weighted formulation enables the network to learn differentiated importance between directly connected joints (e.g., shoulder-elbow) and distant but semantically related joints (e.g., hand-foot coordination patterns). The graph convolution operation processes these structured pose representations through learnable transformations that respect the skeletal topology. For each temporal frame, the GCN applies normalized spectral convolution to propagate information between connected joints:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H_t^{(l+1)} = D^{-\frac{1}{2}}(A + I)D^{-\frac{1}{2}}P_t^T W^{(l)}, \end{aligned}$$\end{document}where the degree normalization matrix D ensures stable information flow across the graph structure, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_t^T \in \mathbb {R}^{J \times 3}$$\end{document} represents the pose coordinates in standard node-feature format, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^{(l)}$$\end{document} contains learnable parameters for layer l. The inclusion of self-connections through the identity matrix I allows each joint to retain its original positional information while incorporating contextual information from neighboring joints. The temporal dimension is handled by independently processing each frame \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t \in \{1, 2,..., T\}$$\end{document} through the graph convolution layers, resulting in a sequence of enhanced joint representations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{H_1^{(l+1)}, H_2^{(l+1)},..., H_T^{(l+1)}\}$$\end{document}. These temporally-distributed features capture both the instantaneous geometric configuration of the human pose and the spatial relationships that define body part coordination. As illustrated in Fig. 3, following graph convolution processing, a residual connection preserves the original pose information while allowing the network to learn additive improvements through structural modeling. Additional convolution operations further refine these enhanced representations to produce the final pose features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h^*$$\end{document}, which encode view-invariant geometric patterns suitable for guiding temporal attention mechanisms.Fig. 3Pose processing module with two tasks, calculating temporal attention weights and pose estimation Pose processing module with two tasks, calculating temporal attention weights and pose estimation Object Detection Module To capture the contextual information essential for distinguishing between activities with similar motion patterns, we incorporate a pre-trained object detection module that identifies and localizes relevant objects within the activity region. The object detection component provides semantic context by recognizing household items, furniture, and manipulated objects that define the activity context, addressing the object interaction complexity challenge inherent in AAL environments. For each video sequence, all frames are processed to identify and localize relevant objects. Rather than using frame-level detections independently, temporal object masks are created that account for object movement throughout the video sequence. For each detected object type, a unified spatial mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_o^{video}$$\end{document} is generated by computing the union of all bounding box regions where the object appears across all cropped frames:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M_o^{video} = \bigcup \limits _{t=1}^{T} M_o^{(t)} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_o^{(t)}$$\end{document} represents the binary mask of object o at frame t, and T is the total number of frames in the video sequence. This temporal aggregation strategy ensures that the spatial attention mechanism can focus on regions where objects are active throughout the entire activity sequence, rather than being limited to instantaneous object positions. To maintain computational efficiency while preserving semantic richness, the dimensionality of object representations is reduced through strategic grouping based on their co-occurrence patterns in indoor scenes. Objects that rarely appear together in the same activity context are grouped together, minimizing information loss while reducing computational overhead for the cross-attention mechanism (see Section 4.4). For each group, a consolidated spatial mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_g^{video}$$\end{document} is generated by computing the union of all individual object masks within that group:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M_g^{video} = \bigcup \limits _{o \in G_g} M_o^{video} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_g$$\end{document} represents the set of objects belonging to group g. This produces group-level spatial masks per video that capture the spatial distribution of semantically related objects while maintaining computational tractability. Each group-level spatial mask is resized to match the spatial dimensions w × h of the I3D feature output, enabling their direct use in creating queries in the cross-attention mechanism. This allows the model to selectively focus on spatial regions where relevant objects are present during activity execution. The object context information guides the spatial attention weights applied to the visual feature maps \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f}$$\end{document} extracted from the I3D network, highlighting the object interaction regions crucial to distinguishing between activities with similar motion patterns but different contextual elements. The integration of object detection with spatial attention mechanisms addresses the fundamental challenge that many indoor activities are defined by the manipulation of specific objects rather than just human motion patterns. By providing explicit object context through spatial masks, the discriminative power of the multi-modal fusion process is enhanced, enabling more accurate fine-grained activity recognition in complex indoor environments. Modality Fusion and Classification To effectively integrate complementary information from human pose and object context modalities with the spatio-temporal visual features extracted by the 3D CNN, a two-stage attention-based fusion mechanism is employed. This design enables dynamic weighting of temporal video features based on pose-derived activity relevance, followed by spatial refinement guided by object interaction context. The resulting fused representation enhances discriminative capacity for fine-grained daily activity recognition in indoor AAL environments. Pose-Driven Temporal Attention Human pose sequences inherently capture view-invariant geometric and dynamic patterns relevant to activity progression. To leverage this modality for temporally adaptive feature weighting, the output of the pose processing GCN is first aggregated and transformed to produce a temporal attention vector that modulates the visual feature map along the temporal dimension. Specifically, as shown in Fig. 3, the pose feature tensor \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H \in \mathbb {R}^{T_p \times J \times C_p}$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_p$$\end{document} is the input temporal length, J is the number of joints, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_p$$\end{document} is the dimension of the pose feature channel, is spatially pooled over the dimension of the joint to obtain temporally ordered pose embeddings:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} h_t = \frac{1}{J} \sum \limits _{j=1}^{J} H_{t,j,:} \in \mathbb {R}^{C_p}, \quad t = 1, \ldots , T_p \end{aligned}$$\end{document}The sequence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{h_t\}_{t=1}^{T_p}$$\end{document} is passed through a multi-layer perceptron (MLP) block producing an output tensor of shape \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_p \times (3 \times J)$$\end{document} to match the dimensions of the original pose tensor P. This dimensional alignment enables a residual connection between the predicted and original pose tensors. This output is then reshaped to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(J \times 3 \times T_p)$$\end{document} and processed by a 1 × 1 convolutional layer that reduces the temporal dimension to 1: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(J \times 3 \times T_p) \rightarrow (J \times 3 \times 1)$$\end{document}, to predict the future pose at time t + Δ , where each prediction yields a pose vector in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{3 \times J}$$\end{document}, constituting an auxiliary pose estimation task. This auxiliary task encourages the pose features to capture meaningful temporal dynamics by learning to forecast subsequent human joint configurations, thereby enhancing the semantic relevance of the temporal attention weights applied to the visual features. To compute temporal attention weights, from the shared MLP layer, a lightweight 1D convolutional layer is first applied to enhance the temporal features. Then, the temporally refined pose features \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h^*$$\end{document} are obtained by first applying temporal pooling that averages every \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n~=~T_p / T$$\end{document} frames, where T is the temporal dimension of the I3D visual feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f} \in \mathbb {R}^{T \times H \times W \times C}$$\end{document}, followed by a subsequent Dense block. The pooling operation reduces the sequence length from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_p$$\end{document} to T to match the temporal dimension of the visual features. These refined features are then used to derive a temporal attention vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha \in \mathbb {R}^T$$\end{document} by normalizing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h^*$$\end{document} via a softmax function to produce a distribution of importance coefficients over the temporal dimension:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \alpha _t = \frac{\exp (h^*_t)}{\sum _{k=1}^T \exp (h^*_k)}, \quad t=1,\ldots ,T \end{aligned}$$\end{document}This attention vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\alpha }$$\end{document} is then applied multiplicatively across the temporal slices of the video feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{f}$$\end{document}, modulating the contribution of each temporal frame according to pose-derived activity relevance:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{\textbf{f}}_{t,:,:,:} = \alpha _t \cdot \textbf{f}_{t,:,:,:}, \quad t=1,\ldots ,T \end{aligned}$$\end{document}By applying pose-driven temporal attention prior to spatial fusion, the framework dynamically emphasizes temporal segments of the video that are more informative for the activity, guided by the intrinsic understanding of the pose modality of human motion dynamics. Object-Guided Spatial Cross-Attention Following temporal weighting, spatial contextualization is performed using grouped object masks derived from the object detection module. These masks serve as semantic queries attending over the temporally modulated visual feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{\textbf{f}}$$\end{document}, enabling selective focus on spatial regions corresponding to relevant object interactions. The 4D feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{\textbf{f}} \in \mathbb {R}^{T \times H \times W \times C}$$\end{document} is reshaped into a flattened spatio-temporal token sequence:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{F} \in \mathbb {R}^{(T \cdot H \cdot W) \times C} \end{aligned}$$\end{document}For each set of the G object groups, a spatial mask is generated by aggregating detections over time and resizing it to match the spatial dimensions of the feature map, resulting in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_g \in \mathbb {R}^{H \times W}$$\end{document}. The mask is then replicated across the T temporal frames to align with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{\textbf{f}}$$\end{document}. Masked average pooling over the spatio-temporal features produces group-level query vectors:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{q}_g = \frac{1}{\sum _{i,j} M_g(i,j) \cdot T} \sum \limits _{t=1}^T \sum \limits _{i=1}^H \sum \limits _{j=1}^W M_g(i,j) \cdot \tilde{\textbf{f}}_{t,i,j,:} \in \mathbb {R}^C \end{aligned}$$\end{document}Each output remains a vector in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^C$$\end{document}, representing a channel-wise summary of the salient features of the corresponding object group. In other words, encoding “what is important” for that group, aggregated over the entire video sequence. Stacking all group queries yields the query matrix:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{Q} = [\textbf{q}_1, \textbf{q}_2, \ldots , \textbf{q}_G]^T \in \mathbb {R}^{G \times C} \end{aligned}$$\end{document}The key and value matrices are set as follows:14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{K} = \textbf{V} = \textbf{F} \in \mathbb {R}^{(T \cdot H \cdot W) \times C} \end{aligned}$$\end{document}and multi-head cross-attention is then applied:15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{A} = \text {MultiHeadCrossAttention}(\textbf{Q}, \textbf{K}, \textbf{V}) \in \mathbb {R}^{G \times C} \end{aligned}$$\end{document}where each output vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{a}_g$$\end{document} encapsulates a contextually attended summary of visual features relevant to object group g.Fig. 4Cross-Attention mechanism between scene context and extracted visual features Cross-Attention mechanism between scene context and extracted visual features Fusion and Final Classification The attended outputs from all object groups are concatenated to form a unified contextual embedding:16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{z}_{context} = [\textbf{a}_1 \Vert \textbf{a}_2 \Vert \ldots \Vert \textbf{a}_G] \in \mathbb {R}^{G \cdot C} \end{aligned}$$\end{document}This embedding is combined with the temporally pooled visual feature vector, obtained by global average pooling over \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{\textbf{f}}$$\end{document}, and concatenated with pose features pooled over time, to form a comprehensive multi-modal representation. The fused vector is then passed through fully connected layers with nonlinear activations and dropout regularization before reaching the final softmax classification layer, which outputs the activity class probabilities. Figure 4 summarizes the process from the attention mechanism to the classification head. Multi-Task Loss Function The network is trained using a weighted multi-task loss function that combines the primary activity classification objective with the auxiliary pose estimation task. The total loss is formulated as:17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{total} = L_{act} + \lambda _{pose} \cdot L_{pose} \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{act}$$\end{document} represents the cross-entropy loss for the main activity classification task, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{pose}$$\end{document} denotes the loss for the auxiliary pose prediction task. The weighting parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{pose}$$\end{document} is tunable and controls the influence of the pose classification task on the overall training objective. The main classification loss is computed as:18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{act} = -\frac{1}{N} \sum _{i=1}^{N} \sum _{c=1}^{C} y_{i,c} \log (\hat{y}_{i,c}) \end{aligned}$$\end{document}where N is the batch size, C is the number of activity classes, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{i,c}$$\end{document} is the ground truth label, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{i,c}$$\end{document} is the predicted probability for sample i and class c. The auxiliary pose loss encourages the temporal attention mechanism to learn meaningful pose dynamics by predicting future joint configurations:19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_{pose} = \frac{1}{N} \sum _{i=1}^{N} \Vert \hat{p}_{i,t+\Delta } - p_{i,t+\Delta }\Vert _2^2 \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{p}_{i,t+\Delta }$$\end{document} is the predicted pose at time t+Δ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{i,t+\Delta }$$\end{document} is the ground truth pose. This multi-task learning approach ensures that the pose-driven temporal attention weights are semantically meaningful and aligned with human motion dynamics, thereby improving the overall activity recognition performance. Discussion on Design Choices The sequential application of pose-driven temporal attention followed by object-guided spatial cross-attention enables the model to first emphasize temporally relevant frames based on intrinsic human motion cues, then spatially refine these frames using contextual object information. This order is justified because temporal weighting benefits from the direct insight of the pose modality into motion dynamics, while spatial attention uses the explicit object layout to discriminate visually similar actions involving different object interactions. Furthermore, applying temporal attention before spatial fusion is mathematically and structurally advantageous. Modulating the visual features temporally applies sharp softmax weights directly across the temporal dimension (T). Reversing this order (applying spatial cross-attention first) would require distributing the temporal weights over a much higher-dimensional, spatially-fused latent space, which severely dilutes the impact of the temporal softmax distribution. Additionally, filtering out irrelevant temporal frames before the spatial cross-attention ensures that the object-guided queries focus only on the active phases of the manipulation, preventing noise from inactive frames from corrupting the spatial attention matrix. The auxiliary pose classification head ensures that the learned temporal attention is semantically meaningful and aligned with the activity recognition objective, mitigating the risk of spurious or noisy attention weights. This multi-task learning approach contributes to improved convergence and robustness. Grouping objects into semantically distinct clusters reduces computational overhead and enforces structured attention, allowing the model to generalize across diverse indoor scenes without overfitting to individual object instances. The architectural decision to provide residual connections only to the auxiliary prediction branch, while excluding direct pose access from the temporal attention mechanism, serves two important purposes. First, this separation enables specialized final processing where each branch can adapt its final representation to its specific objective, while still benefiting from shared temporal feature learning that improves both tasks; the auxiliary prediction benefits from direct pose residuals for future pose estimation, while the attention branch can focus on learning higher-level temporal patterns. Second, this design prevents information leakage where the temporal attention might become overly dependent on direct pose coordinates rather than learning meaningful temporal patterns that generalize across different pose configurations and viewpoints. This approach ensures that the auxiliary pose prediction task fulfills its primary role of enhancing the shared representations for the main activity recognition objective, while allowing each branch to specialize in its final processing steps. Overall, the proposed fusion mechanism addresses key challenges in indoor activity recognition (temporal variability, object interaction complexity, and inter-class similarity) by leveraging complementary modalities through principled attention operations. Experimental Setup Dataset Experiments have been conducted on the Toyota SmartHome dataset [10], which comprises 16,115 video clips of daily living activities performed by 18 senior participants aged 60-80 years. The dataset provides multi-modal data including RGB video (original resolution 640 × 480), depth information, and 3D skeletal pose data captured using seven Kinect v1 cameras positioned across three indoor scenes: dining room, living room, and kitchen. All activities were fully unscripted with participants unaware of the study purpose, ensuring naturalistic behavior patterns. To the best of our knowledge, this is the most suitable dataset for our focus on daily activity recognition among older adults in AAL contexts, as it includes ADLs recorded from multiple camera viewpoints. Pre-processing and Temporal Alignment To handle the multi-modal temporal alignment, input dimensions of 224 × 224 pixels are utilized after human-centered cropping, with the I3D network processing 64-frame sequences while the GCN processes 32-frame pose sequences using 13 joints with 3D coordinates. Temporal alignment between modalities is achieved through uniform sampling, where pose frames are sampled at every second frame to match the temporal coverage of the video sequence. Backbones and Features The I3D network with Inception backbone utilizes pre-trained weights from the Kinetics-400 dataset, with features extracted from the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$8 \times 7 \times 7 \times 1024$$\end{document} dimensional layer before global average pooling. The GCN employs two graph convolution layers processing each pose frame independently, with hidden dimensions of 128 and 64 units respectively, adjacency matrix weights α =5 for connected joints and β =2 for disconnected joints, and ReLU activation functions. Object Detection and Grouping Object detection is performed using YOLOv8 medium variant trained on the COCO dataset with confidence threshold of 0.5. From the COCO vocabulary we retain 40 home-relevant classes. A naïve design would allocate one spatial mask per object (40 masks), but this inflates memory/compute in the fusion module and increases supervision ambiguity when multiple masks activate simultaneously. To control complexity, we reduce the masks to eight by grouping objects. We first considered room-based grouping (e.g., “kitchen”), but this placed many co-activating items into the same mask (e.g., sink, microwave, knife), which hindered discrimination between distinct actions occurring within the same room. Instead, we adopt a few-coincidences strategy that merges objects that rarely appear in the same activities, yielding eight masks that are less likely to co-activate while preserving informative spatial cues. This grouping process is as follows: Let be the set of videos and the set of activities. For each object \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$o_i$$\end{document} and video v, define the object–video incidence\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Z_{i,v} \in \{0,1\}, \quad Z_{i,v}=1 \iff o_i \text { is detected at least once in } v. $$\end{document}For each activity , let be the videos of activity a. We form activity-wise object countsso row a of S describes how often object i appears in activity a. To compare objects independent of their marginal frequency, we normalise each object column over activities:Column \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\cdot ,i}$$\end{document} is the empirical distribution of object i across activities. We measure how differently two objects distribute over activities using the Pearson correlation of their columns, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{corr}(P_{\cdot ,i},P_{\cdot ,j})$$\end{document}. Low (or negative) correlation indicates few coincidences across activities. We start with 40 singleton groups, one for each object. At each step we merge the pair \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(G_p,G_q)$$\end{document} with the smallest correlation. For any group G, we define group–video presence by an n-ary OR\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ Z_{G,v} = \bigvee _{o_i \in G} Z_{i,v}, $$\end{document}and rebuild S and P at the group level: andWe then recompute the correlation matrix over the current groups and repeat until eight groups remain. For reproducibility, the resulting groups are:G1: Knife, Vase, Hair DrierG2: Sink, Orange, Dining TableG3: Remote, Pizza, Mouse, Broccoli, ToasterG4: Refrigerator, Cup, Cake, MicrowaveG5: TV, Hot Dog, Fork, Toilet, Wine Glass, Donuts, Bed, ToothbrushG6: Bench, Banana, Couch, Spoon, AppleG7: Chair, Bowl, LaptopG8: Book, Sandwich, Keyboard, Bottle, Carrot, Cell Phone, Oven, Scissors G1: Knife, Vase, Hair Drier G1: Knife, Vase, Hair Drier G2: Sink, Orange, Dining Table G2: Sink, Orange, Dining Table G3: Remote, Pizza, Mouse, Broccoli, Toaster G3: Remote, Pizza, Mouse, Broccoli, Toaster G4: Refrigerator, Cup, Cake, Microwave G4: Refrigerator, Cup, Cake, Microwave G5: TV, Hot Dog, Fork, Toilet, Wine Glass, Donuts, Bed, Toothbrush G5: TV, Hot Dog, Fork, Toilet, Wine Glass, Donuts, Bed, Toothbrush G6: Bench, Banana, Couch, Spoon, Apple G6: Bench, Banana, Couch, Spoon, Apple G7: Chair, Bowl, Laptop G7: Chair, Bowl, Laptop G8: Book, Sandwich, Keyboard, Bottle, Carrot, Cell Phone, Oven, Scissors G8: Book, Sandwich, Keyboard, Bottle, Carrot, Cell Phone, Oven, Scissors Fusion For each clip we compute one binary mask per group by the temporal union of its member-object masks. The cross-attention fusion uses these 8 masks as queries and employs 8 attention heads to capture diverse object-guided spatial patterns. Training Details The network is trained using the Adam optimizer with a learning rate of 0.001, batch size of 16, and 100 epochs. Dropout regularization with rate 0.3 is applied to prevent overfitting, and early stopping is used with patience of 10 epochs based on validation accuracy. The multi-task loss weighting parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{pose}$$\end{document} is set to 0.25, with pose prediction horizon of 3 frames ahead for the auxiliary task. Training is carried out on a Tesla A100 GPU with 40GB memory using TensorFlow framework, requiring approximately 30 hours of training time. Evaluation Protocols We follow the original Cross-Subject (CS) and Cross-View (CV) protocols [10] for evaluation. In the CS setting, 11 subjects are used for training and 7 for testing across 35 activity classes, including drinking, cooking, movement, and daily living tasks. In the CV setting, the CV1 and CV2 protocols are adopted with 19 activity classes, where different camera combinations are used for training, validation, and testing to assess view-invariant performance. Performance is measured using mean per-class accuracy, following the established Toyota SmartHome evaluation procedures. Results are reported separately for the CS, CV1, and CV2 scenarios. To validate the effectiveness of the proposed multi-modal fusion approach, comparisons are made against various single- and multi-modal baselines. Results and Discussion This section presents a comprehensive evaluation of the proposed multi–modal daily activity recognition framework on the Toyota SmartHome dataset, following the CS, CV1, and CV2 evaluation protocols described in Section 4. We first report the overall performance of the proposed method in comparison with representative single- and multi-modal approaches. This is followed by detailed ablation studies analyzing the contribution of each modality and fusion component, qualitative visualization of the learned attention mechanisms, and an error analysis highlighting the strengths and limitations of the method. Overall Performance Table 1 summarizes the mean per-class accuracy of the proposed system and the compared baseline methods under the three evaluation protocols. The results indicate that the proposed framework consistently outperforms the individual single-modal baselines (video-only, pose-only) as well as conventional fusion strategies, demonstrating the advantage of the proposed method.Table 1Comparison of mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols.∘ indicates that the modality has been used only in trainingMethodsPoseRGBCSCV1CV2Pose only 2s-AGCN [23]\checkmark × 60.921.632.3 ST-GCN [7]\checkmark × 53.812.551.0 UNIK [4]\checkmark × 63.122.961.4 MS-G3D Net [44]\checkmark × 66.717.559.4 ViA [45]\checkmark × 64.035.665.4RGB only I3D [3]× \checkmark 53.434.945.1 AssembleNet++ [9]× \checkmark 63.6–– LTN [46]× \checkmark 65.9–54.6 VPN++ [8]∘ \checkmark 69.0–54.9 π -ViT [28]∘ \checkmark 72.955.264.8 SV-data2vec [29]∘ \checkmark 72.946.057.5RGB + Pose P-I3D [47]\checkmark \checkmark 54.235.150.3 Separable STA [10]\checkmark \checkmark 54.235.250.3 VPN [31]\checkmark \checkmark 65.243.854.1 VPN++ + 3D Poses [8]\checkmark \checkmark 71.0–58.1 MMNet [34]\checkmark \checkmark 70.137.446.6 π -ViT + 3D Poses [28]\checkmark \checkmark 73.155.665.0Ours\checkmark \checkmark 70.144.265.4Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Table 1Comparison of mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols.∘ indicates that the modality has been used only in trainingMethodsPoseRGBCSCV1CV2Pose only 2s-AGCN [23]\checkmark × 60.921.632.3 ST-GCN [7]\checkmark × 53.812.551.0 UNIK [4]\checkmark × 63.122.961.4 MS-G3D Net [44]\checkmark × 66.717.559.4 ViA [45]\checkmark × 64.035.665.4RGB only I3D [3]× \checkmark 53.434.945.1 AssembleNet++ [9]× \checkmark 63.6–– LTN [46]× \checkmark 65.9–54.6 VPN++ [8]∘ \checkmark 69.0–54.9 π -ViT [28]∘ \checkmark 72.955.264.8 SV-data2vec [29]∘ \checkmark 72.946.057.5RGB + Pose P-I3D [47]\checkmark \checkmark 54.235.150.3 Separable STA [10]\checkmark \checkmark 54.235.250.3 VPN [31]\checkmark \checkmark 65.243.854.1 VPN++ + 3D Poses [8]\checkmark \checkmark 71.0–58.1 MMNet [34]\checkmark \checkmark 70.137.446.6 π -ViT + 3D Poses [28]\checkmark \checkmark 73.155.665.0Ours\checkmark \checkmark 70.144.265.4Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Comparison of mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols.∘ indicates that the modality has been used only in training Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Under the CS protocol, our approach achieves 70.1% accuracy, which is competitive with recent transformer-based state-of-the-art methods while using a much lighter architecture, providing practical implementation advantages. While π -ViT achieves superior performance (72.9% CS), it employs a heavy transformer-based architecture with TimeSformer backbone, requires large-scale pretraining dataset, and utilizes dual auxiliary modules (2D-SIM and 3D-SIM) with separate token-skeleton mapping and feature alignment tasks. Similarly, SV-data2vec (72.9% CS) employs a computationally intensive transformer architecture with self-supervised pretraining using teacher-student framework requiring exponential moving average updates and complex masked prediction training procedures. Remarkably, our multi-modal architecture demonstrates that domain-tailored computationally efficient methods can effectively bridge the performance gap with resource-intensive transformer architectures while maintaining significant computational advantages. Furthermore, within the CNN-GCN architectural category, our approach significantly outperforms existing methods, including P-I3D, Separable STA, and VPN, by substantial margins in all the protocols, confirming superior fusion design. Notably, on the CV2 protocol, our method exhibits superior view-invariant capabilities (65.4%), surpassing both π -ViT (64.8%) and SV-data2vec (57.5%), thus rivaling transformer-based approaches in challenging cross-view scenarios. System Component Evaluation To validate the effectiveness of each component in the proposed multi-modal framework, a comprehensive study on the Toyota SmartHome dataset is conducted across all three evaluation protocols. This analysis examines the contribution of individual modalities, fusion strategies, preprocessing techniques, and architectural design choices.Table 2Comprehensive ablation study results showing mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocolsConfigurationCSCV1CV2Single-modal baselines Video-only (I3D)53.434.945.1 Pose-only (GCNs)52.142.651.2Bi-modal baselines Video + Pose68.442.762.1 Video + Object66.839.560.3Multi-modal baselines Full system70.144.265.4 w/o pose normalization67.840.162.9 Semantic coherence grouping68.842.862.7Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Table 2Comprehensive ablation study results showing mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocolsConfigurationCSCV1CV2Single-modal baselines Video-only (I3D)53.434.945.1 Pose-only (GCNs)52.142.651.2Bi-modal baselines Video + Pose68.442.762.1 Video + Object66.839.560.3Multi-modal baselines Full system70.144.265.4 w/o pose normalization67.840.162.9 Semantic coherence grouping68.842.862.7Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Comprehensive ablation study results showing mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols Bold values are used to highlight (i) the results obtained with the proposed method and (ii) the best performance within each design combination. This visual emphasis is intended to facilitate comparison across configurations Table 2 illustrates a modality effectiveness study. The single-modal baselines establish the foundation for understanding each modality’s contribution. Video-only demonstrates stronger cross-subject performance compared to other metrics, but suffers significantly in cross-view scenarios, particularly CV1, highlighting view variance challenges. Pose-only shows more consistent performance across protocols due to its inherent view-invariant properties. The bi-modal combinations reveal complementary relationships between modalities. Video + Pose achieves the most significant improvement over individual modalities, demonstrating that geometric information effectively complements visual features for person-independent recognition. Video + Object provides moderate improvements, with object context helping disambiguate activities with similar motion patterns. The full multi-modal system achieves optimal performance across all protocols, validating the fusion architecture effectiveness. Removing pose normalization results in notable performance degradation, particularly in cross-view scenarios, confirming the critical importance of view-invariant preprocessing for robust indoor activity recognition. This analysis confirms that each component serves a distinct and essential role in the multi-modal framework, with the full system achieving a balanced integration of all three modalities for optimal AAL activity recognition performance. To assess the effectiveness of our grouping strategy, we compared it against a baseline that forms groups without considering cross-activity co-occurrence (i.e., ignoring how often objects appear together in the same videos). Our few-coincidences approach, merging objects that rarely co-occur across activities, achieved higher accuracy while operating under the same computational budget (eight masks). By minimising within-group co-activation, it reduces redundancy and yields more discriminative group-level representations, better capturing the variety of object–activity interactions in daily living. We also study the effect of the number of attention heads in the cross-attention module (Table 3), in line with standard Transformer practice. Increasing heads from 2 to 8 yields consistent gains, with the best average performance at 8 heads. Using 16 heads offers no consistent improvement, while incurring additional compute. We therefore adopt 8 heads as a balanced choice.Table 3Performance comparison across different numbers of attention heads showing accuracy results on CS, CV1, and CV2 datasetsNumber of headsCSCV1CV22 heads69.243.564.84 heads69.443.265.18 heads70.144.265.416 heads69.843.965.6 Table 3Performance comparison across different numbers of attention heads showing accuracy results on CS, CV1, and CV2 datasetsNumber of headsCSCV1CV22 heads69.243.564.84 heads69.443.265.18 heads70.144.265.416 heads69.843.965.6 Performance comparison across different numbers of attention heads showing accuracy results on CS, CV1, and CV2 datasets Figure 5 illustrates the learned temporal attention weights alongside corresponding video frames, demonstrating how our pose-driven temporal attention mechanism focuses on activity-relevant temporal segments. The visualization shows that the attention mechanism successfully identifies key motion phases, such as the preparation, execution, and completion stages of activities like “Eating Snacks,” confirming the effectiveness of our pose-guided temporal weighting strategy.Fig. 5Sample frames from the intervals corresponding to temporal attention weights \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _t$$\end{document} values for activity class “Eating Snacks” Sample frames from the intervals corresponding to temporal attention weights \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _t$$\end{document} values for activity class “Eating Snacks” Error Analysis and Limitations To better understand the proposed method’s constraints, we analyzed the confusion matrices for both the CV and CS protocols. The analysis of the Cross-Subject (Fig. 6) protocol shows that the proposed multi-modal system achieves stronger performance on activities with clear whole-body motion and distinctive temporal structure. Classes such as Walk, WatchTV, Sitdown, Getup, and Leave obtain some of the highest correct classification rates. This suggests that the 3D CNN and GCN streams are effective when the activity can be recognized from global movement patterns and body posture. Under the CS protocol, this is important because the model is evaluated on subjects not seen during training, so it must learn activity representations that generalize across different people.Fig. 6Confusion matrix for cross-subject protocol (in %) Confusion matrix for cross-subject protocol (in %) The results also show that intra-class variability remains a central challenge. In the CS setting, the same activity may be performed with different speed, posture, range of movement, or object-handling style depending on the person. This is visible in errors involving daily activities that can be carried out in several ways, such as Eat.Attable, which is confused with Drink.Fromcup and Readbook. These errors suggest that, for some classes, the model may capture the general body configuration or scene context but struggle to learn a subject-independent representation of the activity itself. A second major source of error is inter-class similarity, especially among actions with comparable hand and arm movements. The matrix shows confusions among cooking-related activities and among activities commonly performed while sitting, such as Readbook, Sitdown, Takepills, Uselaptop, and Usetablet. These actions are semantically different, but their visual and skeletal patterns are close. This indicates that the model can still confuse activities when the main difference lies in the object used or in a subtle change in motion direction. The object detection module is intended to help with fine-grained ambiguity by capturing human–object interaction cues. However, the confusion matrix suggests that object information does not always resolve these errors. One likely reason is the limited resolution of the visual input. Although the original Toyota SmartHome RGB videos have a resolution of 640 × 480, the frames are cropped and downsampled before being processed. This reduction in spatial detail can make small household objects, such as cups, glasses, cans, bottles, and kettles, difficult to distinguish. Objects with similar shape, size, color, or position may therefore appear visually similar to the model. This also affects the pre-trained YOLO detector, which may struggle to differentiate fine-grained items under these conditions. As a result, the object stream may provide useful contextual information, but not always sufficiently reliable object-specific cues when the activity label depends on subtle object differences. Enhancing the input video quality, preserving higher-resolution crops around hands and objects, or using a more robust high-resolution object detector could help reduce this bottleneck and improve recognition of object-centered activities. Overall, the CS results show that the proposed multi-modal framework generalises well for activities defined by clear body motion, but struggles with fine-grained activities affected by intra-class variability, inter-class similarity, and object interaction complexity. This is consistent with the challenges of indoor AAL environments, where different people perform the same daily activity in different ways. The results support the value of combining video, skeleton, and object streams, while also showing that future improvements should strengthen hand–object relation modeling and subject-invariant representations for subtle daily activities.Fig. 7Confusion matrices for cross-view protocol (in %) Confusion matrices for cross-view protocol (in %) The CV1 and CV2 confusion matrices (Fig. 7) show how the proposed multi-modal system behaves under cross-view evaluation. In both protocols, the test camera is different from the training camera or cameras, so the main difficulty is view variance: the same activity may appear with different body orientation, object visibility, hand trajectories, occlusions, and spatial layout. The contrast between CV1 and CV2 is clear. CV1 is much more unstable, while CV2 shows a substantial improvement for most classes. This is expected because CV2 uses more training views, giving the model more visual and pose variability from which to learn. In CV1, the system performs well for a limited set of activities. Eat.Attable reaches a very high correct classification rate (96.9%), followed by Sitdown (89.6%), Leave (78.6%), and Getup (77.9%). These classes have strong contextual or postural cues. For instance, Eat.Attable is strongly linked to the table scene, while Sitdown, Getup, and Leave involve clearer body transitions. This suggests that the 3D CNN and GCN streams can still capture robust global motion and pose information when the action has a distinctive structure. However, CV1 performs poorly for many fine-grained activities. Drink.Frombottle, Drink.Fromcan, Drink.Fromglass, Pour.Frombottle, Readbook, and Uselaptop show low correct classification rates. These results indicate that training from a single viewpoint does not provide enough variability for the model to generalize reliably to the test view, especially when the action depends on small objects, hand motion, or subtle posture differences. A key issue in CV1 is the over-prediction of Eat.Attable and Sitdown. Several activities are classified as these activities. This suggests that, under strong viewpoint change, the model may rely too heavily on table-related context and seated posture. When the object is small, occluded, or poorly resolved, the cross-attention mechanism may prioritize scene and pose features over object-specific cues. CV2 shows a much stronger pattern of recognition. Many classes improve markedly compared with CV1. These improvements show the benefit of training with several viewpoints. The model is exposed to more variation in appearance, pose, and object visibility, so the learned features become more view-invariant. Even so, CV2 does not remove all errors. Some inter-class similarity remains, especially between object-based actions. Drinking activities are still confused, and Pour.Frombottle is confused with Drink.Frombottle. These errors suggest that the model recognizes the general action family, such as drinking or manipulating a container, but still struggles to distinguish fine object identity or interaction intent. This is particularly relevant for AAL applications, where many activities of daily living differ mainly in the object used and how it is used. Overall, the comparison between CV1 and CV2 shows that the proposed framework benefits greatly from multi-view training. CV1 reveals the vulnerability of the model when learning from a restricted viewpoint, with strong over-reliance on scene context and posture. CV2 demonstrates that additional camera views improve view-invariant learning and substantially reduce many errors, particularly in drinking, eating, and device-use activities. Nevertheless, the remaining confusions show that inter-class similarity, object interaction complexity, low-resolution object cues, and view-dependent occlusion still limit performance. Future improvements should therefore focus on higher-resolution hand–object regions, stronger temporal modeling of object use, and more robust fusion strategies that can better adapt to viewpoint changes. Computational Complexity and Real-World Deployment Given that AAL systems are intended for continuous, real-time monitoring of older adults, the computational complexity and inference latency of the deployed architecture are critical considerations. While the proposed multi-modal framework achieves competitive accuracy, its constituent modules have vastly different computational footprints. Analysis of the architecture reveals that the core action recognition and fusion components (the I3D visual backbone, the GCN for pose processing, and the cross-attention module) are computationally efficient. The GCN and attention mechanisms account for a negligible fraction of the total parameters (under 5 million combined) and operations. The primary computational bottleneck resides in the frame-by-frame feature extraction, specifically the object detection module (YOLOv8m) and, in pure RGB deployments, the requisite 3D human pose estimator (e.g., MediaPipe [48] or HRNet [49]). Together, these frame-level spatial extractors account for an estimated 85% to 90% of the total Floating Point Operations (FLOPs) required to process a 64-frame video clip. To achieve real-time latency on the limited hardware typical of smart home frameworks (e.g., edge AI devices or local home servers), several mitigation strategies can be done. First, it is important to note that in modern smart home ecosystems, object detection and human pose tracking are foundational, shared services utilized by multiple applications (such as security monitoring, fall detection, and automation). Therefore, the computational cost of these modules is often amortized across the entire smart home infrastructure rather than being exclusively borne by the activity recognition system. Furthermore, the framework’s deployment feasibility can be significantly enhanced through standard edge optimization techniques. The object detector can be scaled down to a lighter and newer variants (e.g., YOLOv12n optimized for edge devices) with minimal impact on coarse object grouping, or configured to run at a lower frame rate, as the spatial layout of background objects rarely changes drastically within a 2-second activity clip. Finally, translating the trained models using hardware-accelerated runtimes (such as NVIDIA TensorRT or ONNX Runtime) and applying model quantization (e.g., reducing weights from FP32 to INT8) can drastically reduce memory bandwidth and inference latency, enabling this holistic multi-modal approach to operate smoothly on resource-constrained edge devices. Conclusion This paper presents an effective multi-modal approach for recognizing daily activities in indoor assisted living environments. By integrating video data, human pose information, and object context through a carefully designed cross-attention mechanism, the system achieves robust and accurate classification despite challenges such as viewpoint variance, inter-class similarity, and complex indoor scenes. The model benefits from the complementary strengths of each modality, allowing it to capture rich semantic and temporal cues critical for fine-grained activity understanding. Experimental results on the benchmark dataset demonstrate consistent improvements over single- and multi-modal baselines as well as recent transformer-based approaches, highlighting the value of leveraging multi-modal fusion with spatial-temporal attention. However, it is important to acknowledge that this framework is explicitly tailored for naturalistic AAL settings. Consequently, our object grouping strategy is purposefully designed for realistic residential environments like those in the Toyota SmartHome dataset, and the system is not intended to generalize to artificial lab-based or outdoor domains. Additionally, the framework’s performance inherently depends on the resolution and accuracy of the underlying spatial extractors, namely the pre-trained pose estimator and object detector. Looking forward, an important research direction is to reduce the dependence on multiple modalities during inference, thereby simplifying deployment and improving efficiency. Developing modality-agnostic representations or knowledge distillation methods that allow the system to operate effectively using only RGB input while retaining performance is a promising avenue. In addition, exploring self-supervised or unsupervised learning techniques could substantially lessen reliance on annotated data and improve the model’s adaptability to new environments and subjects. Further improvements might come from designing adaptive attention mechanisms that dynamically allocate computational resources based on scene complexity or activity context, balancing performance and efficiency. Investigating domain adaptation and lifelong learning strategies would also advance the system’s generalization to diverse real-world conditions. Moreover, future work on the object modality could focus on more efficient grouping and attention mechanisms to better capture the spatial and temporal relationships between humans and relevant objects without significantly increasing computational costs. With respect to transformer architectures, while recent advances show promise, challenges remain concerning their high computational demand and data requirements. Developing hybrid models that integrate transformer-based attention within lighter multi-modal frameworks, combined with pretraining strategies, may yield more practical and effective systems for assisted living scenarios. By addressing these directions, future work can contribute to more scalable, privacy-aware, and intelligent activity recognition solutions that meet the needs of real-world assisted living scenarios.