Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations

David Schneider, Simon Reiß †, Marco Kugler †, Alexander Jaus †, Kunyu Peng †, Susanne Sutschet †, M. Saquib Sarfraz, Sven Matthiesen †, Rainer Stiefelhagen † · 2024

Read Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations by David Schneider, Simon Reiß †, Marco Kugler †, Alexander Jaus †, Kunyu Peng †, Susanne Sutschet †, M. Saquib Sarfraz, Sven Matthiesen †, Rainer Stiefelhagen † (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00128.txt.

Description

Abstract Exploring the intricate dynamics between muscular and skeletal structures is pivotal for understanding human motion. This domain presents substantial challenges, primarily attributed to the intensive resources required for acquiring ground truth muscle activation data, resulting in a scarcity of datasets.

Catalog record

CREATORDavid Schneider, Simon Reiß †, Marco Kugler †, Alexander Jaus †, Kunyu Peng †, Susanne Sutschet †, M. Saquib Sarfraz, Sven Matthiesen †, Rainer Stiefelhagen †
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations David Schneider, Simon Reiß †, Marco Kugler †, Alexander Jaus †, Kunyu Peng †, Susanne Sutschet †, M. Saquib Sarfraz, Sven Matthiesen †, Rainer Stiefelhagen † Abstract Exploring the intricate dynamics between muscular and skeletal structures is pivotal for understanding human motion. This domain presents substantial challenges, primarily attributed to the intensive resources required for acquiring ground truth muscle activation data, resulting in a scarcity of datasets. In this work, we address this issue by establishing Muscles in Time (MinT) , a large-scale synthetic muscle activation dataset. For the creation of MinT , we enriched existing motion capture datasets by incorporating muscle activation simulations derived from biomechanical human body models using the OpenSim platform, a common approach in biomechanics and human motion research. Starting from simple pose sequences, our pipeline enables us to extract detailed information about the timing of muscle activations within the human musculoskeletal system. Muscles in Time contains over nine hours of simulation data covering 227 subjects and 402 simulated muscle strands. We demonstrate the utility of this dataset by presenting results on neural network-based muscle activation estimation from human pose sequences with two different sequence-to-sequence architectures. Data and code are provided under https://simplexsigil.github.io/mint . 1 Introduction Like prisoners in Plato’s cave, neural networks for human motion understanding often rely on indirect representations rather than direct, biologically grounded data. In Plato’s allegory, prisoners in a cave see only shadows cast on the wall, not the true objects. Similarly, neural networks trained on accessible data, such as RGB and depth-based video recordings or motion capture, only perceive surface-level appearance of motion in contrast to the inner mechanics of the human body. This reliance on external visual observations provides an incomplete understanding of the true complexities of human motion. Just as the prisoners lack a direct view of the objects casting the shadows, current models lack exposure to the internal workings of the human body, such as the muscle activations driving motion. This gap limits their ability to develop an in-depth understanding of physical exertion, motion difficulty, and mass impact on the body. Our community has progressed from capturing human motion with camera sensors and predicting activities to pose-based recognition systems that account for the body and its motion over time. These advances, while significant, still overlook the interplay of muscle activations, which are the root of pose sequences and patterns. Figure 1: Simulation pipeline of the Muscles in Time dataset. The SMPL representation is extracted from videos, then, the SMPL represented motions are mapped to bio-mechanically validated human body models to simulate fine-grained muscle activation, connecting computer vision with biomechanical research. Bottom right : two activation sequences for exemplary muscles. Images from [ 47 , 15 ] . Collecting electromyographic (EMG) data or more commonly used surface electromyographic (sEMG) data, as a measure of muscle activation, presents challenges. It is resource intensive, requiring specialized equipment, controlled environments, and is an invasive procedure. Existing EMG and sEMG datasets are small, limited in scope, and not representative of the variety of human motions. These limitations hinder the development of neural networks that can generalize across different types of motion and subjects. While acknowledging the contributions of EMG and sEMG datasets, we identify an opportunity to supplement this domain with a synthetic dataset that overcomes some limitations of real-world data collection. The strength of our dataset lies in its scale and detail of muscle activation data, a feat not achievable through conventional methods alone. Every dataset, simulated or real, has domain-specific fidelity and relevance. Real-world recordings offer authenticity that underpins our understanding of human biomechanics with nuances, such as EMG measurements being subject-specific and varying over the course of one day. Simulated datasets, like ours, offer a complementary perspective by providing comprehensive data for the understanding of muscle activation patterns through a scalable data acquisition pipeline. In this work, we present a comprehensive large-scale dataset incorporating muscle activation information. We enrich existing motion capture datasets with muscle activation simulations from biomechanical models of the human body. Our pipeline uses simple pose and shape sequences with estimated weight and mass of the human body to simulate muscle activations for individual movements. Using this, we generate the muscles’ activation that fit the provided human motions. Figure 1 provides an overview of our pipeline. We showcase the utility of muscle activations as an additional data type for human motion understanding and gather insights by visualizing the intricate details of our data. Our dataset, the first of its magnitude and detail, describes muscle activation across a wide array of movements. By enhancing the current set of tools available to researchers, we expand the potential for scientific investigation and innovation in the study of human motion. 2 Related Work Human Motion Analysis and Datasets EMG-based muscle activation analysis is a well-established field in biomechanical research. Still, publicly available databases including experimentally measured muscle excitation using sEMG are often small in size or cover a small range of muscles or motion variations [ 21 , 30 , 79 , 25 , 51 , 46 , 61 , 42 , 34 , 50 , 65 ] . The dataset proposed by Zhang et al . [ 79 ] contains 5 persons and leveraged 8 EMG sensors. The KIMHu dataset [ 30 ] , for example, includes sEMG data of four upper limb muscles measured during different arm exercises performed by 20 subjects. The MIA Dataset [ 10 ] includes sEMG signals for eight muscles in total (upper and lower limb) across 10 subjects who performed 15 different exercises, e.g., running, jumping jacks, squats, and elbow punches. MuscleMap is a video-based muscle activation estimation dataset, which assigns binary muscle activation labels to action categories, involving 20 muscle groups and 135 actions [ 55 ] . In Table 1 , we provide a comparison of multiple recent EMG datasets to MinT. Most notably MinT features a significantly larger number of subjects, a larger number of activation measurements and a diverse range of motions. OpenSim is an open-source software platform for musculoskeletal modeling, simulation, and analysis. It is widely used in various research areas such as biomechanics research, orthopedics and rehabilitation science, and medical device design [ 16 , 66 ] . The state-of-the-art process in OpenSim for simulating muscle activations of a certain task requires subject-specific motion and force data. In most cases, these data are obtained through experimental studies, which can be time-consuming and resource-intensive. In a related field, musculoskeletal humanoid control and simulation focuses on developing computational models and control strategies for simulating human motion with musculoskeletal detail. Recent work by Jiang et al. [ 35 ] , Caggiano et al. [ 4 ] , Feng et al. [ 23 ] , and He et al. [ 83 , 29 ] has advanced methods for efficient and realistic simulation of muscle-actuated characters. While these approaches differ from OpenSim’s focus, they highlight the broader interest in understanding and simulating human musculoskeletal dynamics. Skeleton-based Vision Models Skeleton-based action recognition [ 22 , 1 ] is pivotal in decoding human actions from video footage, providing a streamlined and insightful depiction of human poses and movements that remains invariant to

Read the complete text