BioHuman: Learning Biomechanical Human Representations from Video

Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu · 2026

Read BioHuman: Learning Biomechanical Human Representations from Video by Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14772.txt.

Introduction

Huo, Zhang, Song, Song, Zuo and Yu introduce BioHuman, a model that reads human motion and muscle activations from monocular video. Progress has been limited by the lack of large data sets with biomechanical annotations. A simulation-based framework first estimates muscle activations from existing motion capture data, yielding BioHuman10M, a large set with synchronised video, motion and activations. The end-to-end model then jointly predicts motion and muscle activity, with claimed accurate reconstruction that generalises across subjects and motions.

Catalog record

CREATORYujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45263
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BioHuman: Learning Biomechanical Human Representations from Video Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu Abstract . Understanding human motion beyond surface kinematics is crucial for motion analysis, rehabilitation, and injury risk assessment. However, progress in this domain is limited by the lack of large-scale datasets with biomechanical annotations, and by existing approaches that cannot directly infer internal biomechanical states from visual observations. In this paper, we introduce a simulation-based framework for estimating muscle activations from existing motion capture datasets, resulting in BioHuman10M, a large-scale dataset with synchronized video, motion, and activations. Building on BioHuman10M, we propose BioHuman, an end-to-end model that takes monocular video as input and jointly predicts human motion and muscle activations, effectively bridging visual observations and internal biomechanical states. Extensive experiments demonstrate that BioHuman enables accurate reconstruction of both kinematic motion and muscle activity, and generalizes across diverse subjects and motions. We believe our approach establishes a new benchmark for video-based biomechanical understanding and opens up new possibilities for physically grounded human modeling. Figure 1. BioHuman10M is a Bio mechanical Human dataset with 10M paired frames, aligning real-world visual motion with physics-based musculoskeletal states. It contains representative action sequences from five datasets, including (a) MotionPRO, (b) EMDB, (c) 3DPW, (d) Human3.6M, and (e) BEDLAM, with synchronized real images, SMPL bodies, and OpenSim musculoskeletal models annotated with ground reaction forces and muscle activations. Teaser. 1. Introduction Understanding human motion beyond surface kinematics from videos is a long-standing goal in computer graphics, with broad applications in animation, simulation, and human-centered analysis. While existing motion capture systems ( 15 ; 24 ; 31 ; 30 ) primarily recover kinematic representations (e.g., SMPL ( 18 ) ), providing an accurate description of how humans move, but offering limited insight into why they move. Extending motion capture ( 14 ; 33 ; 23 ) to infer underlying biomechanical states, such as muscle activations, is a natural next step towards more complete human modeling. This opens up new opportunities for applications in rehabilitation, sports analysis, human–computer interaction, and assistive technologies such as exoskeleton systems. The first challenge towards biomechanical understanding from video is the lack of large-scale datasets that jointly provide vision, human motion, and biomechanical annotations such as muscle activations. Existing datasets ( 9 ; 19 ; 23 ) are often incomplete across modalities, making it difficult to bridge visual observations, human motion, and biomechanics. Collecting these datasets requires optical motion capture systems, EMG sensors, and expert supervision, making such real-world datasets scarce and difficult to acquire at scale. In addition, the sparsity of wearable sensors makes full-body muscle activation capture challenging ( 4 ) . To address these challenges, we construct BioHuman10M, a large-scale dataset featuring synchronized video, human motion, and muscle activations, enabling direct supervision of biomechanical human representations. The dataset is built upon widely used motion capture datasets, including Human3.6M ( 9 ) , 3DPW ( 29 ) , BEDLAM ( 2 ) , EMDB ( 12 ) , and MotionPRO ( 21 ) . We propose BioSim ( Bio mechanics Sim ulation), a simulation framework that estimates muscle activations from SMPL motion sequences based on OpenSim ( 5 ) , to construct BioHuman10M, a large-scale dataset that bridges vision, human motion, and biomechanics. Beyond the data limitation, existing approaches typically adopt a two-stage pipeline that first reconstructs kinematic motion and subsequently estimates muscle activations through biomechanical simulation. OpenCap ( 27 ) and OpenCap Monocular ( 6 ) first estimate markers or SMPL ( 18 ) from videos, and then perform OpenSim-based simulation to infer biomechanics. MinT ( 23 ) replaces the OpenSim simulation stage with a learned neural approximation, thereby improving inference speed. VID ( 3 ) estimates joint torques directly from images, but does not model full-body muscle activations or musculoskeletal states. Such a decoupled formulation is inherently limited, as errors introduced during motion reconstruction inevitably propagate to downstream biomechanical inference, increasing both modeling complexity and optimization difficulty. Our key insight is that existing methods treat motion reconstruction and muscle activation estimation as decoupled problems, despite human motion being fundamentally governed by the musculoskeletal system. In practice, motion and muscle activations are tightly coupled and should be inferred jointly, allowing biomechanical cues and motion observations to mutually benefit from each other. This motivates a unified framework for joint motion and biomechanical modeling. Building on BioHuman10M, we propose BioHuman, an end-to-end framework that directly predicts human motion and muscle activations from monocular video sequences. Given a monocular video as input, BioHuman employs PromptHMR ( 30 ) as a front-end encoder to extract both SMPL pose estimates and image-conditioned HMR features, which are further processed by a trainable temporal transformer. The pose stream provides explicit kinematic trajectories, while the visual stream preserves frame-level image evidence such as appearance, orientation, and pose uncertainty. By jointly reasoning over these complementary cues, BioHuman simultaneously models human motion and biomechanics within a unified framework. Our dataset and method advance motion capture from geometric motion estimation toward deeper musculoskeletal understanding. Beyond recovering body pose, BioHuman predicts the underlying muscle activation patterns that drive human movement. Extensive experiments demonstrate the effectiveness of our framework. In summary, our contributions are as follows: • We introduce BioHuman10M, a large-scale dataset that augments existing motion capture datasets with biomechanically simulated muscle activations, enabling paired supervision of video, motion, and muscle signals. • We propose BioHuman, an end-to-end framework that directly predicts human motion and muscle activations from monocular video. • Extensive experiments demonstrate the effectiveness of our method, showing that jointly modeling human motion and muscle activations significantly improves biomechanical prediction accuracy. 2. Related work 2.1. Physiologically Plausible Motion Capture Existing motion capture systems primarily recover geometric and kinematic representations of human motion from visual observations. The emergence of parametric human body models, such as SMPL, has led to significant progress in monocular human pose and shape capture. These approaches ( 11 ; 34 ; 35 ) model the human body through surface geometry and articulated joint motion, enabling robust motion reconstruction while offering limited understanding of the physiological and biomechanical mechanisms underlying human movement. To improve realism and physiological plausibility, subsequent works ( 1 ; 17 ; 20 ) introduced anatomical and physics-related constraints into motion capture pipelines. MANIKIN ( 10 ) explicitly incorporates joint angle limits into motion capture to regularize reconstructed motions and avoid anatomically implausible poses. Meanwhile, human representations themselves are gradually evolving from purely geometric and kinematic descriptions towards physiologically and biomechanically grounded modeling. Anatomically informed body models such as OSSO ( 14 ) , BOSS ( 25 ) , and SKEL ( 13 ) further incorporate explicit skeletal structures into human representations, enabling more realistic modeling of bones and articulations. Building upon SKEL, HSMR ( 33 ) further demonstrates physiologically plausible motion capture through anatomically constrained skeletal modeling. Despite these advances, existing approaches primarily improve motion plausibility through external constraints and anatomical priors, but lack the ability to directly infer how the musculoskeletal system generates motion. In particular, internal biomechanical states such as muscle activations remain largely unexplored in vision-based motion capture. Table 1. Comparison between BioHuman10M and representative related datasets (M. A.: Muscle Activations). Dataset Frames Image Motion GRF M. A. Human3.6M 3.6M AMASS \sim 10M

Read the complete text