Semantically Encoding Activity Labels for Context-Aware Human Activity Recognition

Wen Ge § , Guanyi Mou § , Emmanuel O. Agu, Kyumin Lee · 2025

Read Semantically Encoding Activity Labels for Context-Aware Human Activity Recognition by Wen Ge § , Guanyi Mou § , Emmanuel O. Agu, Kyumin Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07916.txt.

Introduction

Wen Ge, Kyumin Lee and their co-authors present SEAL, a method for context-aware human activity recognition in 2025. Past methods treated activity labels as plain binary tags, which misses the rich links between similar activities like walking and running.

SEAL uses language models to encode activity and context labels as vectors that keep their meaning links. Sensor data from smart devices are encoded into the same shared space during training. At test time, the system picks the label whose vector lies closest to the input data. The authors say this helps when labels are noisy or when the best sensor type is missing.

Catalog record

CREATORWen Ge § , Guanyi Mou § , Emmanuel O. Agu, Kyumin Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantically Encoding Activity Labels for Context-Aware Human Activity Recognition Wen Ge § , Guanyi Mou § , Emmanuel O. Agu, Kyumin Lee Abstract Prior work has primarily formulated Context-Aware Human Activity Recognition (CA-HAR) as a multi-label classification problem, where model inputs are time-series sensor data and target labels are binary encodings representing whether a given activity or context occurs. These CA-HAR methods either predicted each label independently or manually imposed relationships using graphs. However, both strategies often neglect an essential aspect: activity labels have rich semantic relationships. For instance, walking, jogging, and running activities share similar movement patterns but differ in pace and intensity, indicating that they are semantically related. Consequently, prior CA-HAR methods often struggled to accurately capture these inherent and nuanced relationships, particularly on datasets with noisy labels typically used for CA-HAR or situations where the ideal sensor type is unavailable (e.g., recognizing speech without audio sensors). To address this limitation, we propose Semantically Encoding Activity Labels (SEAL), which leverage Language Models (LMs) to encode CA-HAR activity labels to capture semantic relationships. LMs generate vector embeddings that preserve rich semantic information from natural language. Our SEAL approach encodes input-time series sensor data from smart devices and their associated activity and context labels (text) as vector embeddings. During training, SEAL aligns the sensor data representations with their corresponding activity/context label embeddings in a shared embedding space. At inference time, SEAL performs a similarity search, returning the CA-HAR label with the embedding representation closest to the input data. Although LMs have been widely explored in other domains, surprisingly, their potential in CA-HAR has been underexplored, making our approach a novel contribution to the field. Our SEAL approach has been rigorously evaluated on three real-world datasets, demonstrating its superior performance. It consistently outperforms state-of-the-art methods by 7.8% to 22.6% in MCC and 3.9% to 8.4% in Macro-F1. Furthermore, SEAL performance is agnostic to the data encoding framework utilized, enhancing performance by 4.7% to 73.3% in MCC and 2.6% to 30.5% in Macro-F1 across different data encoding models. This robust performance opens up new possibilities for integrating more advanced LMs into CA-HAR tasks. We share our code at https://github.com/GMouYes/SEAL . § § footnotetext: Equal contribution I Introduction Context-Aware Human Activity Recognition (CA-HAR) is a critical task that involves detecting human activities and their contexts using sensor data from devices such as smartphones and smartwatches [ 1 , 2 ] . Traditionally, CA-HAR has been formulated as a multi-label classification task, where the inputs are time-series sensor data, and the target labels are binary 0/1 encodings that indicate the absence/presence of specific activities and contexts. Previous research (Fig. 1a ) has focused on creating models that map sensor data to binary labels [ 3 , 4 , 5 ] . More recent work has explored modeling label co-occurrences using manually defined, learned graphical representations to capture some relationships between labels [ 6 , 7 , 8 ] . (a) Traditional approach. (b) Our approach. Fig. 1: Comparison of traditional Machine Learning approach v.s. our multi-modality alignment approach. Traditional approaches directly map labels into binary values. In contrast, our approach leverages the language model to encode the semantic relationship between context and activities within high-dimensional vector representations and leads to better performance. However, these approaches overlook a key consideration: activity labels have intrinsic, nuanced semantic relationships that are not captured when labels are assigned independent binary values, or missed by manually defined graphical relationships. For example, activities such as Walking , Jogging , and Running share similar movement patterns but differ in pace and intensity, suggesting that they are semantically close. On the other hand, misclassifying Trembling as Standing due to subtle signal changes could lead to missed early signs of medical conditions such as Parkinson’s disease [ 9 ] . Current models struggle to represent these subtle relationships, leading to suboptimal CA-HAR performance. To bridge the gap, we propose a novel framework called Semantically Encoded Activity Labels ( SEAL ), a unique approach that leverages Language Models (LMs) [ 10 ] and their associated rich embedding spaces to encode the rich semantics inherent in activity labels (Fig. 1b ). This novel approach is a departure from traditional methods and is likely to pique the interest of researchers and practitioners in the field. LMs have become foundational in Natural Language Processing (NLP) due to their ability to transform natural language into vector representations that preserve semantics. From a machine learning perspective, CA-HAR can be re-envisioned as an alignment task between two modalities: time-series data (sensor signals) and text (activity and context labels). During training, SEAL aligns the vector embeddings of sensor data with the embeddings of their corresponding activity/context labels. At inference time, it performs a similarity search to determine the label whose encoding is close to the input sensor data. By leveraging LMs and their associated embedding space with strong semantics, SEAL preserves semantic relationships between activity labels, capturing nuanced relationships between activities and contexts. Our work aligns with trending research in other domains [ 11 , 12 ] where text-image alignments gained increasing momentum and enabled various real-world applications [ 13 , 14 ] . (a) Other HAR models using Language Models (LMs). (b) Our design. Fig. 2: Comparison of other HAR models using Language Models v.s. our design. While other approaches mainly use LMs as auxiliary components that provide guidance without directly participating in the decision-making process, our approach integrates LM as a primary contributor, allowing its active involvement in the activity recognition task. Although prior work has explored incorporating LMs into CA-HAR [ 15 , 16 ] , these efforts primarily leveraged the reasoning capabilities of LMs by using them as auxiliary modules that refine the outputs of CA-HAR models but did not integrate them as a core module in an end-to-end framework (see Fig 2 ). In contrast, our SEAL framework integrates LMs as a central component, serving as the primary encoder of activity labels. This innovative SEAL framework addresses the critical issue of semantic information loss inherent in binary label encodings, a significant limitation of prior CA-HAR methods. Furthermore, our SEAL approach achieves robust performance improvements and generalizes effectively, even on the relatively noisy datasets commonly used in CA-HAR. In this paper, we define context-aware human activity the as the < activity, phone placement > tuple. In summary, our contributions are as follows: \bullet We identify the loss of semantic relationship information caused by binary label encodings widely used in prior CA-HAR frameworks. We reframe CA-HAR as a time-series-to-text alignment task in a shared vector embedding space, a perspective that should enlighten our audience about the nuanced relationships between activity labels. \bullet We introduce SEAL , a novel framework that, for the first time, leverages language models to semantically encode activity labels and align these textual encodings with sensor data representations, improving activity recognition. \bullet We extensively evaluate SEAL , comparing it to five state-of-the-art baselines on three real-world CA-HAR datasets. Our detailed evaluations provide robust evidence of SEAL ’s performance, demonstrating that it consistently outperforms baseline approaches, with MCC/Macro-F1 improvements ranging from 7.8%/3.9% to 22.6%/8.4%. This comprehensive evaluation should provide reassurance to our audience about the effectiveness of our approach. \bullet We provide rigorous analysis and statistical

Read the complete text