Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets

Daniel Geißler, Dominique Nshimyimana, Vitor Fortes Rey, Sungho Suh, Bo Zhou, Paul Lukowicz · 2024

Read Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets by Daniel Geißler, Dominique Nshimyimana, Vitor Fortes Rey, Sungho Suh, Bo Zhou, Paul Lukowicz (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09037.txt.

Introduction

Daniel Geißler and co-authors dig in this 2024 paper into the data sets that grade human activity recognition, the field that teaches machines to read body motion. Scores on six popular bench sets stall well below the near-perfect marks seen in twin fields, and the authors ask if the data, not the models, is at fault. They run six top methods and gather the samples every one of them gets wrong.

That shared fail set exposes real flaws: fuzzy labels, glitches in recording, and smeared borders between activities. The team counts and sorts the faults, offers a three-way mask so others can patch the sets, and urges cleaner collection habits for the next round of benchmarks.

Catalog record

CREATORDaniel Geißler, Dominique Nshimyimana, Vitor Fortes Rey, Sungho Suh, Bo Zhou, Paul Lukowicz
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09037

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets Daniel Geißler, Dominique Nshimyimana, Vitor Fortes Rey, Sungho Suh, Bo Zhou, Paul Lukowicz Abstract . The research of machine learning (ML) algorithms for human activity recognition (HAR) has made significant progress with publicly available datasets. However, most research prioritizes statistical metrics over examining negative sample details. While recent models like transformers have been applied to HAR datasets with limited success from the benchmark metrics, their counterparts have effectively solved problems on similar levels with near 100% accuracy. This raises questions about the limitations of current approaches. This paper aims to address these open questions by conducting a fine-grained inspection of six popular HAR benchmark datasets. We identified for some parts of the data, none of the six chosen state-of-the-art ML methods can correctly classify, denoted as the intersect of false classifications (IFC). Analysis of the IFC reveals several underlying problems, including ambiguous annotations, irregularities during recording execution, and misaligned transition periods. We contribute to the field by quantifying and characterizing annotated data ambiguities, providing a trinary categorization mask for dataset patching, and stressing potential improvements for future data collections. DOI: XXXXXXX.XXXXXXX Journal: IMWUT Volume: 0 0 0 0 CCS: Human-centered computing Ubiquitous and mobile computing design and evaluation methods 1. Introduction A significant amount of research on machine learning (ML) methods for sensor-based human activity recognition (HAR) has emerged over the past decade with the growing availability of public datasets, especially based on inertial measurement units (IMUs). The datasets such as PAMAP2 ( Reiss and Stricker, 2012 ) and Opportunity ( Chavarriaga et al., 2013 ) have accelerated machine learning (ML) development as ML research, especially deep learning, can focus on the model architectures and be independent from the time-consuming data collection processes ( Wang et al., 2019 ; Bian et al., 2022 ) . They also allow a fair comparison across different proposed machine learning models as benchmarks. However, most such ML-focused works only conclude by reporting the statistical level metrics and confusion matrix and fall short of examining the details, especially the negative samples. However, in typical studies evaluating the viability of novel sensing modalities, such negative samples are crucial for the discussion. Some recent ML models proposed for the HAR datasets are adapted from architectures that have achieved high levels of recognition in datasets from other disciplines with similar classification tasks. For example, for text classification, transformer models like DeBERTa and C-BERT ( Karl and Scherp, 2022 ) have reached over 98% accuracy in the 8-class subset of the Reuters-21578 dataset (R8) 1 1 1 http://www.daviddlewis.com/resources/testcollections/reuters21578/ ; vision transformers like ViT-H/14 ( Dosovitskiy et al., 2021 ) have reached over 99% accuracy in the popular 10-class image classification dataset CIFAR-10 ( Krizhevsky et al., 2009 ) . Although there are other benchmarks that have inferior reported results, like the ImageNet dataset ( Deng et al., 2009 ) currently topping at 92% by OmniVec ( Srivastava and Sharma, 2024 ) , there are 1000 classes of objects which is inherently more challenging than 10-class problems as the statistical chance level is significantly lower. Yet, the adapted versions of these models like TinyHAR ( Zhou et al., 2022 ) which is a transformer-based model still struggle in many HAR classification benchmarks with similar or fewer numbers of classes (e.g. PAMAP2 at the range of 70 to 80%). This is especially true when proper leave-out partitions between the training, validation, and testing data were performed in combination with cross-validation to avoid overfitting. The nature of different modalities can partially account for this difference, as wearable sensor signals are obscure while vision or text are easily interpretable by human perceptions, requiring advanced mechanism to enhance the data and model insight ( Geißler et al., 2023b ) . Nonetheless, the obscure nature of sensor signals essentially leaves researchers little device for verifying the data context but to have full faith in the provided annotations as ground truth. Yet human annotators inevitably make mistakes or produce ambiguities as pointed out by Hoelzemann, et al. ( Hoelzemann et al., 2024 ) As shown in Fig. 1 , combining the lack of attention given to negative samples, inferior performance from better-established modalities, and human factors in annotation, we raise the research question: ‘Are some of the falsely classified samples in the benchmark datasets inherently ambiguous, making it impossible to achieve 100% accuracy?’ Figure 1 . We aim to examine HAR research beyond the typical approach focusing on the statistical metrics and confusion matrices tested on public benchmark datasets to demonstrate improvement. There have been discussions in this direction in the literature, including developing remedies. Ward, et al. ( Ward et al., 2011 ) pointed out the influence of granular discrepancies between the prediction and annotations on the short time window level, and proposed event-based metrics. Kwon et al. ( Kwon et al., 2019 ) attempted to address the annotation uncertainty problem by artificially inserting a Gaussian distribution as a smoothing kernel to each consecutive sequence of the same class. A recent study by Hoelzemann, et al. ( Hoelzemann et al., 2024 ) revealed the human factors in HAR dataset annotation. Despite these works attempting to overcome the inherent annotation ambiguities in HAR, there has been no systematic analysis that quantifies the extent, characterizes the underlying problems, and provides viable solutions for existing datasets, which is the main motivation of our work. To answer such questions and provide solutions to help move HAR forward, we conducted a fine-grained inspection of the datasets in this work. This paper first provides a literature overview of the popular HAR benchmarks and how they have helped accelerate machine learning development in this field in the past two decades. Then we use the most widely used ML methods which can be considered milestones in the related work (five model architectures as well as pre-training enhancement) to examine the datasets, especially the samples that cannot be successfully detected by any of the models. We look beyond the summary of statistical metrics and inspect the dataset on the granularity of seconds in short windows. We have found that there is a significant overlap of the false classifications among all of the models in all of the datasets. We then closely examine those data windows that none of the models can correctly classify, and attempt to characterize the underlying problem. Detailed examination of these common false classifications reveals several common phenomena: (1) Ambiguous annotations: in a single-output classification task, the different classes should be mutually exclusive. However, we have found some classes can be composed of other classes, or different classes have common elements of motion. (2) Execution during recording: in some classes that are supposed to be static, the participant may have short windows of transient spontaneous random motions. This in combination with annotation of large granularity (i.e. only annotating the start and end of long segments of activities) usually results in irregularities that are incoherent with the designated labels. (3) Transition periods: Depending on the labeling design granularity, transitions between activity classes are commonly falsely classified if the transition period is excessively long and inconsistently labeled. In the wake of these identified and characterized problems in the six datasets, we

Read the complete text