Portable biomechanics laboratory enables clinically accessible movement analysis from a handheld smartphone Authors: J. D. Peiffer, Kunal Shah, Irina Djuraskovic, Shawana Anarwala, Kayan Abdou, Rujvee Patel, Prakash Jayabalan, Brenton Pennicooke, R. James Cotton Published: 2026 ABSTRACT Movement directly reflects neurological and musculoskeletal health, yet objective biomechanical assessment is rarely available in routine care. We introduce Portable Biomechanics Laboratory (PBL), a platform for fitting biomechanical models to handheld smartphone video. We validate PBL on over 15 hours of data synchronized to ground truth motion capture, finding joint-angle errors < 3∘ across patients with neurological injury, lower-limb prosthesis users, pediatric inpatients, and controls. Across 1021 videos recorded in prospective clinical deployment, PBL was easy to implement, yielded reliable gait metrics (ICC > 0.9), and detected clinically relevant differences in movement. For cervical myelopathy patients, its gait quality measures correlated with modified Japanese Orthopedic Association (mJOA) scores and were responsive to clinical intervention. Handheld smartphone video can therefore deliver accurate, scalable, and low-burden biomechanical measurement, enabling greatly increased monitoring of movement impairments. We release the first clinically validated method for measuring whole-body kinematics from handheld smartphone video at https://IntelligentSensingAndRehabilitation.github.io/MonocularBiomechanics/. Introduction Many clinical conditions have pronounced movement phenotypes which are rarely measured in clinical practice1–3. For example, temporal parameters, including stride time and swing time, have been shown to predict fall risk in older adults4. Patients with knee osteoarthritis (KOA) also show altered gait kinematics, which can predict disease progression5,6. Gait kinematics can also characterize recovery dynamics after neurological injury and even predict responses to particular interventions7,8. The Stroke Recovery and Rehabilitation Roundtable emphasized the importance of incorporating movement quantification into clinical trials, while noting that logistical challenges in capturing such data remain a significant barrier9. Traditionally, gait analysis requires a specialized laboratory equipped with an optical motion capture (OMC) system and force plates, making it expensive, time-consuming, and typically only covered by insurance in limited circumstances10. In some cases, such as orthopedic surgical planning, force plate measurements and electromyography are also required to ensure the precision of marker-based measurements. Currently, gait is more commonly characterized using the 10 m walk or 6 min walk tests, which capture only speed and endurance. Other assessments-such as the Timed Up and Go (TUG)11, the Berg Balance Scale (BBS)12, and the Functional Gait Assessment (FGA)13-evaluate movement quality using clinical scoring methods that are either subjective or limited to stopwatch-based timing, but do not directly quantify the movement itself. Scalable, accessible movement analysis in the clinic would substantially improve upon the status quo. Recent advances in AI-based methods offer a promising path forward. Multi-view markerless motion capture (MMMC) requires much less time to acquire and less manual post-processing than OMC. In recent years, numerous studies have demonstrated that MMMC produces similar kinematics to OMC in multiple populations14–23. We have developed methods for MMMC24–27 using MuJoCo28, a high-performance physics simulator for machine learning that supports differentiable biomechanical models. This enables us to directly end-to-end optimize the kinematics and skeleton scale from videos. Our end-to-end approach outperforms alternatives that reproduce multi-stage marker-based pipelines that first estimate virtual marker locations and then compute inverse kinematics on these trajectories25,26. Although MMMC is faster than OMC, installing and calibrating several high-performance cameras remains a significant barrier to clinical implementation. OpenCap substantially lowers the technical burden of MMMC by using two or three tripod-mounted smartphones, achieving < 5 degrees of joint-angle error relative to OMC29 and detecting movement patterns that classify disease23,30. While OpenCap is a significant and widely used contribution, the requirement for two tripods placed at ± 45 degrees is not always feasible in clinical settings, which often include small exam rooms and tight hallways. In these contexts, a single handheld camera offers a more practical alternative. Monocular video analysis has been applied clinically in Parkinson’s disease31, cerebral palsy32, and stroke33 populations. However, these approaches infer 2D or 3D joint locations, or non-biomechanically-grounded mesh representations, rather than biomechanical joint angles used in traditional movement analysis. Recent methods34,35 have explored data-driven biomechanical reconstruction from a single camera, but have not been trained or validated on clinical populations. To enable data capture in real-world clinical settings, we previously developed a smartphone application for collecting handheld videos in clinics36, which we refer to as the Portable Biomechanics Laboratory (PBL), and validated gait event detection from these recordings37. We subsequently demonstrated our end-to-end biomechanical pipeline can reconstruct biomechanically-grounded kinematics from handheld smartphone video, with or without wearable sensors, achieving accurate knee kinematics from video alone38. The central goal of the present study is to evaluate whether this smartphone-based biomechanics pipeline can produce accurate kinematics across diverse patient populations, movement tasks, and clinical environments that show construct validity against clinical metrics. We perform this validation by first evaluating our system against OMC in control populations, next against MMMC in clinical populations, and finally, our intended use case, collect data in actual clinic hallways to compare against clinical scores (Fig. 1). Overall, we find PBL has high joint-angle accuracy compared to OMC and MMMC, concordance with clinical scores, and is responsive to clinical intervention. Together, these evaluations demonstrate that quantitative movement analysis can be integrated into routine clinical workflows using a single handheld smartphone for accurate and clinically relevant biomechanical measurement.Fig. 1Study design overview.We first validated the Portable Biomechanics Laboratory (PBL) against an optical motion capture dataset in which control participants performed a range of upper- and lower-limb activities. We then evaluated PBL in clinical populations by recording participants performing lower-limb tasks using both multi-camera markerless motion capture and PBL in hospital laboratories and on clinical floors. Finally, we deployed PBL in clinic hallways and therapy gyms, recording participants after clinical visits to assess the relationship between PBL-derived gait quality scores and standard clinical measures. Illustrations by Kristina Moore. ⓒ2026 Kristina Moore. Used with permission under a CC BY 4.0 license. Study design overview. We first validated the Portable Biomechanics Laboratory (PBL) against an optical motion capture dataset in which control participants performed a range of upper- and lower-limb activities. We then evaluated PBL in clinical populations by recording participants performing lower-limb tasks using both multi-camera markerless motion capture and PBL in hospital laboratories and on clinical floors. Finally, we deployed PBL in clinic hallways and therapy gyms, recording participants after clinical visits to assess the relationship between PBL-derived gait quality scores and standard clinical measures. Illustrations by Kristina Moore. ⓒ2026 Kristina Moore. Used with permission under a CC BY 4.0 license. Results Ground truth comparison In general, kinematics from a single camera closely matched MMMC and OMC. Example overlays demonstrate our method is robust to common mobility aids (Fig. 2) and kinematic traces (Fig. 3a) reflect clinical impairments like neurologic or prosthetic asymmetries. Quantitative ground truth comparisons are presented in Table 1 and Supplement.Fig. 2Biomechanical Reconstructions from Handheld Smartphone.Using only a handheld smartphone, our PBL and end-to-end biomechanical fitting approach robustly captures movement across varied clothing types, assistive devices, and clinical environments. a We first validate this approach on clinical outpatients in our lab, including participants using assistive devices or receiving support from a clinician as needed. b We next deploy this approach in an outpatient clinic for patients with gait impairments, finding this method minimally disrupting clinical workflow while capturing relevant gait features. c This technique allows deployable biomechanical capture in dynamic clinical settings as well as outdoors.Fig. 3Quality measures of single camera fitting.a Kinematic traces from smartphone video (red/blue) compared to ground truth (gray dashed) during walking. b Joint angle errors across populations for select lower limb angles. c Select joint angle errors with respect to camera view angle show that sagittal plane angles have the lowest error with sagittal camera views, and frontal angles have the lowest error with frontal views.Table 1Summary of errors for different datasetsMMMCOMCHandheldStaticStaticMedian Absolute Error (deg)2.79 (0.86)2.95 (0.77)2.75 (0.48)Mean Absolute Error (deg)4.10 (1.68)4.40 (1.62)3.89 (1.66)RTE (cm)6.44 (4.77)5.00 (4.68)2.41 (0.63)Median and Mean Absolute Errors are for joint angles aggregated across all joints, participants, and trials. RTE is the median root translation error (in cm). Errors are presented as Median (nIQR) or Mean (std). Biomechanical Reconstructions from Handheld Smartphone. Using only a handheld smartphone, our PBL and end-to-end biomechanical fitting approach robustly captures movement across varied clothing types, assistive devices, and clinical environments. a We first validate this approach on clinical outpatients in our lab, including participants using assistive devices or receiving support from a clinician as needed. b We next deploy this approach in an outpatient clinic for patients with gait impairments, finding this method minimally disrupting clinical workflow while capturing relevant gait features. c This technique allows deployable biomechanical capture in dynamic clinical settings as well as outdoors. Quality measures of single camera fitting. a Kinematic traces from smartphone video (red/blue) compared to ground truth (gray dashed) during walking. b Joint angle errors across populations for select lower limb angles. c Select joint angle errors with respect to camera view angle show that sagittal plane angles have the lowest error with sagittal camera views, and frontal angles have the lowest error with frontal views. Table 1Summary of errors for different datasetsMMMCOMCHandheldStaticStaticMedian Absolute Error (deg)2.79 (0.86)2.95 (0.77)2.75 (0.48)Mean Absolute Error (deg)4.10 (1.68)4.40 (1.62)3.89 (1.66)RTE (cm)6.44 (4.77)5.00 (4.68)2.41 (0.63)Median and Mean Absolute Errors are for joint angles aggregated across all joints, participants, and trials. RTE is the median root translation error (in cm). Errors are presented as Median (nIQR) or Mean (std). Summary of errors for different datasets Median and Mean Absolute Errors are for joint angles aggregated across all joints, participants, and trials. RTE is the median root translation error (in cm). Errors are presented as Median (nIQR) or Mean (std). For the MMMC dataset, the Median Absolute Error between a handheld moving camera and MMMC was 2.79∘. This error was slightly higher for static camera fits using a single view from the MMMC system (2.95∘), likely due to the greater distances to the person. The error was the greatest in the neurological population, with a Median Absolute Error of 3.32∘, lower for LLPUs (2.97∘), and lowest for controls at 2.51∘ (Supplementary Fig. 1, Supplementary Table 2). Our method also performed well on the BML-MoVi dataset, showing only 2.75∘ of Median Absolute Error from the monocular fits compared to OMC. Error distributions are shown for lower limb joints in Fig. 3b and broken down across all joints in Supplementary Table 1, Supplementary Table 2 and Supplementary Fig. 1. Notably wide error distributions are Neck Extension, Lumbar Extension, Hip Flexion (static camera), with third quartiles above 5∘. Frontal-plane joint angles (e.g., hip adduction) were best estimated from frontal views, while sagittal-plane joint angles (e.g., hip and knee flexion) were more accurately estimated from sagittal views (one-sided Mann-Whitney U-tests; p < 0.005 for all comparisons). This difference was greatest for knee flexion at 1.02∘ (Fig. 3c). Visual comparisons of root position between single-camera and multi-view fits are shown in Supplementary Fig. 2. The clinical dataset using a handheld, rotating smartphone yielded an average RTE of 6.44 cm, while a static camera yielded 5.00 cm. RTE for the BML-MoVi dataset with a static smartphone averaged 2.41 cm (Table 1). Clinical feasibility PBL was accessible and easy to use, causing minimal disruption to routine workflows in both KOA and CM clinics. It required no special clothing, lighting, or calibration-we simply followed participants with a handheld phone as they walked. Clinical inference from smartphone-based kinematics Extracting valid, repeatable, and responsive measurements from our smartphone-based system is arguably more important than accuracy alone. We computed the GDI as a measure of gait quality, testing it for repeatability, validity and responsiveness (Fig. 4, Table 2). We also test for differences in raw kinematic traces pre/post surgery using the paired SPM1D t-tests39,40.Fig. 4Clinical Validity of Smartphone-Based Gait Deviation Index.a Hip and Knee flexion angles of clinical and control groups. b GDI separates groups at risk of falls determined by the Berg Balance Scale. c GDI correlates with 10 Meter Walk Test performance r = 0.82. d GDI of LLPUs and KOA participants is significantly lower than that of control populations. Further, GDI of Transfemoral amputees is significantly lower than GDI of Transtibial amputees. e GDI collected in clinical settings correlates (r = 0.47) with the mJOA, a clinically used ordinal questionnaire.Table 2Reliability, validity, and responsiveness of smartphone-based metricsRepeatabilityValidityResponsivenessICC2ICC2kr mJOAr mJOA-LEEffect SizePatient-ReportedmJOA————0.24 (−0.35 0.76)mJOA-LE————0.22 (−0.41 0.91)Smartphone-BasedCadence0.58 (0.52 0.64)0.88 (0.85 0.90)0.25*0.39*0.39 (−0.09 0.77)DST0.41 (0.35 0.48)0.78 (0.72 0.82)−0.27*−0.37*−0.52 (−0.84 −0.11)*GDI0.84 (0.81 0.86)0.96 (0.96 0.97)0.47*0.48*0.44 (−0.02 0.76)*ICCs are presented as ICC (95% confidence intervals). mJOA and mJOA-LE show Spearman correlations between gait metric and ordinal ratings. Wilcoxon Effect Size for changes pre/post surgery presented with 95% confidence intervals. * indicates p < 0.05. Clinical Validity of Smartphone-Based Gait Deviation Index. a Hip and Knee flexion angles of clinical and control groups. b GDI separates groups at risk of falls determined by the Berg Balance Scale. c GDI correlates with 10 Meter Walk Test performance r = 0.82. d GDI of LLPUs and KOA participants is significantly lower than that of control populations. Further, GDI of Transfemoral amputees is significantly lower than GDI of Transtibial amputees. e GDI collected in clinical settings correlates (r = 0.47) with the mJOA, a clinically used ordinal questionnaire. Table 2Reliability, validity, and responsiveness of smartphone-based metricsRepeatabilityValidityResponsivenessICC2ICC2kr mJOAr mJOA-LEEffect SizePatient-ReportedmJOA————0.24 (−0.35 0.76)mJOA-LE————0.22 (−0.41 0.91)Smartphone-BasedCadence0.58 (0.52 0.64)0.88 (0.85 0.90)0.25*0.39*0.39 (−0.09 0.77)DST0.41 (0.35 0.48)0.78 (0.72 0.82)−0.27*−0.37*−0.52 (−0.84 −0.11)*GDI0.84 (0.81 0.86)0.96 (0.96 0.97)0.47*0.48*0.44 (−0.02 0.76)*ICCs are presented as ICC (95% confidence intervals). mJOA and mJOA-LE show Spearman correlations between gait metric and ordinal ratings. Wilcoxon Effect Size for changes pre/post surgery presented with 95% confidence intervals. * indicates p < 0.05. Reliability, validity, and responsiveness of smartphone-based metrics ICCs are presented as ICC (95% confidence intervals). mJOA and mJOA-LE show Spearman correlations between gait metric and ordinal ratings. Wilcoxon Effect Size for changes pre/post surgery presented with 95% confidence intervals. * indicates p < 0.05. GDI separated groups with expected differences; for instance GDI was significantly lower for participants scoring higher fall risk on the BBS and Transfemoral LLPUs had a significantly lower GDI than Transtibial LLPUs (Fig. 4b, d). The GDI also correlates well with the 10MWT (Pearson’s r = 0.82), one of the most commonly measured clinical outcomes for gait (Fig. 4c). Finally, the GDI showed a significant correlation with the BBS (r = 0.62), mJOA (r = 0.47) and the lower extremity subscore of the mJOA (r = 0.48) (Fig. 4e, Table 2). We analyzed the > 13,500 steps across 222 individuals recorded with the PBL system and found high repeatability of 0.84 for the GDI of a single step (ICC2) and higher repeatability when averaging multiple steps together: 0.96 (ICC2k) (Table 2). ICCs for spatiotemporal metrics (cadence, DST) were lower—likely due to timing variability—but remained highly repeatable when averaged (ICC2k). Smartphone-derived gait metrics (GDI and DST) demonstrated significant postoperative changes in individuals with cervical myelopathy (Wilcoxon signed-rank test, p < 0.05), whereas patient-reported clinical scales (mJOA and mJOA-LE) did not show significant change (Table 2). Wilcoxon effect sizes were 0.24 for mJOA, 0.22 for mJOA-LE, -0.52 for DST, and 0.44 for GDI. In addition, paired SPM1D t-tests revealed a significant postoperative increase in knee flexion during initial swing (64–68% of the gait cycle). Discussion We present a clinically validated method for extracting biomechanically accurate kinematics from smartphone video in clinical settings. Across diverse populations, our single-camera system showed strong agreement with ground truth (typically < 3∘ Median Absolute Error). Gait quality metrics derived from these videos detected clinically relevant group differences and were responsive to clinical intervention. These findings suggest that a clinician, therapist, or medical assistant could feasibly perform gait analysis in under a minute to obtain an objective measurement of a patient’s gait impairment to track their recovery. Joint angle errors were generally low across all populations and settings (Table 1). We attribute the slightly higher Median Absolute Error in clinical populations (0.46 - 0.81 degrees, Fig. 3b, Supplementary Fig. 1b, Supplementary Table 2) to assistive devices such as canes, walkers, ankle foot orthoses, or gait belts (Fig. 2) and the fact that keypoint detection does not perform as well on prosthetic and orthotic devices36. Studies using OpenCap also report increases in joint angle errors in clinical populations22. Intuitively, sagittal-plane angles were most accurate when recorded from sagittal viewpoints, and frontal-plane angles were most accurate from frontal viewpoints (Fig. 3c), giving clinicians a simple and actionable strategy for mitigating joint-angle error during data collection. Our only dataset comparing PBL to Optical Motion Capture (OMC) is BML-MoVi, which only contains able-bodied controls. To further validate the biomechanical generalization of PBL, we also included a dataset with diverse etiologies of gait impairments assessed with our previously validated MMMC system25,41. Although MMMC differs from OMC, multiple independent studies have demonstrated strong agreement between the two approaches14,16–21,27. Nonetheless, neither method constitutes a true biomechanical gold standard relative to biplanar fluoroscopy, due to known limitations including marker placement variability42 and soft-tissue artifact43. The consistently strong correspondence observed between PBL and both OMC and MMMC across a diverse participant population supports the biomechanical accuracy and generalizability of the proposed approach (Table 1). While monocular video-based analysis is expected to exhibit some loss of accuracy relative to multiview systems, it substantially improves accessibility in clinical environments. Thus, the latter part of this study evaluates whether PBL collected in clinical settings is sufficiently accurate to demonstrate construct validity against established clinical metrics. To demonstrate this, we showed that movement quality scores from PBL: (1) change with expected population changes, (2) correlate with clinical scores, and (3) are responsive to clinical intervention. First, GDI from PBL was decreased for inpatients with fall risk assessed by the BBS and further decreased for inpatients classified as high fall risk (Fig. 4b). Similarly, GDI was decreased for Transtibial LLPUs and further decreased for Transfemoral LLPUs, an intuitive result. Second, GDI correlated highly with the 10MWT (Fig. 4c), a widely used clinical outcome measure. The GDI also had a moderate correlation with the mJOA (Fig. 4e, Table 2), which is expected considering the GDI is a continuous quantitative measure and the mJOA is a patient-reported ordinal scale. Regarding responsiveness, SPM1D analysis of kinematic traces showed significant changes in knee flexion following surgical intervention, establishing that our system can be used to analyze specific changes in movement and quantitatively track recovery. Quantitative metrics (DST and GDI) extracted from our smartphone-based system showed significant postoperative changes, whereas the mJOA did not (Table 2). This may suggest that smartphone-derived kinematic measures capture postoperative kinematics changes not reflected in patient-reported scores, but it is important to highlight these are measuring fundamentally different constructs. Thus, quantitative gait metrics (e.g., GDI, DST) complement patient-reported outcomes such as the mJOA, and we defer to future studies to investigate whether they may precede and causally influence patient outcomes44. We focused on the GDI because it is a widely used summary of gait kinematics. However, we do not claim it is the optimal metric for every application. Instead, we anticipate that scalable gait analysis will enable new research to identify the most clinically meaningful measures. For example, we recently described a method for inferring torques and ground reaction forces from kinematics and saw that the ground reaction forces are sensitive to clinical history45. The minimally clinically important difference for kinematic features of gait has not yet been established for many clinical conditions. While our accuracy is generally quite high, greater accuracy may be required for certain indications. For example, the 5 degrees of error we typically see at the ankle might be insufficient to detect subtle changes in plantarflexion spasticity or toe clearance from a single camera. We observe heavy-tailed error distributions, with some trials exceeding 10∘ of error (Supplementary Fig. 1c). Although occasional large errors and failure cases warrant further investigation, these errors occur infrequently and do not introduce systematic bias. Aggregated kinematic trajectories and summary metrics remain strongly correlated with clinically relevant measures (Table 2, Fig. 4), supporting the method’s reliability for population-level analysis and leaving subject-level clinical decision-making to future work. We tested whether fitting residuals-2D reprojection error or 3D keypoint error-could serve as reliability metrics. Their correlations with joint-angle error were weak (0.35 and 0.24, respectively; Supplementary Fig. 5), underscoring the need for better confidence estimates in markerless motion capture. We have previously developed calibrated confidence intervals for monocular pose estimation without biomechanics46 and for multi-view biomechanical estimation24; extending these ideas to monocular biomechanics is an important direction for future work. Other approaches have proposed deployable systems for human motion capture. Most well-known is OpenCap29, which uses two or three smartphones on static tripods to triangulate joint locations from video and fit biomechanics using the widely used OpenSim framework47,48. OpenCap reports a Mean Absolute Error of 4.1∘ during walking in 10 controls using two static cameras. In our control cohort (n = 29), PBL achieves a comparable Mean Absolute Error of 4.32 (2.06)∘ using a single handheld moving camera. While this error was computed on similar populations and activities, we note this is not a direct comparison and leave that to future work. While more cameras will generally increase accuracy24, our results suggest that the difference between OpenCap with 2 camera on tripods and a single handheld smartphone is now small. Apart from accuracy, OpenCap and our PBL have different advantages: OpenCap provides a very user friendly platform, uses the widely accepted OpenSim package47, and calculates joint kinetics. Our PBL approach presented here is markedly easier to deploy in clinical settings such as narrow hallways, small exam rooms, and busy gym spaces. The same simplicity also enables potential use in patients’ homes with recordings performed by caregivers. Extending our method to include joint kinetics is feasible with future work such as ref. 45. While our pipeline is not real-time, per-trial runtimes are compatible with same-day or visit-level clinical analysis. Two recent approaches estimate biomechanics from monocular video: BioPose34 and Human Skeleton and Mesh Recovery (HSMR)35. Both regress pose directly from images, making them faster than our pipeline. However, neither was trained or evaluated on clinical populations, so their performance on clinical gait-often far outside the distribution of healthy movement (Fig. 4a)-is uncertain. Our method instead uses trajectory optimization with in-loop biomechanical model scaling applied consistently across trials, aligning more closely with established biomechanical reconstruction practices. It can also integrate wearable sensors38 to address occlusion and high-speed motion. For healthy controls, our kinematic accuracy is similar to BioPose (< 3.9∘ Mean Absolute Error on MoVi dataset), though the comparison is incomplete because BioPose does not specify which joints contribute to its reported errors. HSMR does not report joint-angle errors. Ultimately, broader development of AI-driven biomechanics tools will expand clinical movement analysis, and large resulting datasets will support data-driven, precision rehabilitation strategies44. In conclusion, we present a method for extracting accurate and biomechanically interpretable joint kinematics from smartphone videos. We validate this method in inpatient, outpatient therapy, and clinical settings, finding it to be highly usable and robust-even in the presence of nearby therapists and assistive devices. Importantly, gait scores extracted from this approach correlate well with commonly used clinical scales such as the 10MWT, BBS, and mJOA, and respond to clinical intervention. This approach enables increased fidelity in clinical and at-home monitoring of movement impairments, paving the way for a big data revolution in movement science. Methods Portable biomechanics laboratory We developed the Portable Biomechanics Laboratory (PBL), a smartphone-based platform that records color video, optional depth video, and internal phone sensor data (magnetometer, accelerometer, gyroscope, and orientation) from a moving, handheld smartphone (Fig. 5a). A gimbal was used only to improve visualization; the method remains accurate without it.Fig. 5Biomechanical reconstruction overview.We introduce a method for biomechanically grounded movement analysis in clinical settings using a handheld smartphone. a Researchers held a smartphone (optionally with gimbal) while following a participant walking. Our system has no specific requirements regarding viewing angle, distance to subject, or therapist assistance. b Recorded smartphone video and optional wearable sensor data are stored in the cloud, and processed using PosePipe, an open-source package implementing computer vision models for person tracking and keypoint detection. c To reconstruct movement, we represent movement as a function that outputs joint angles, which–combined with body scaling parameters and evaluated through forward kinematics–generate a posed biomechanical model in 3D space. This untrained model is compared to video-extracted joint locations and optionally smartphone sensor data to compute a loss. This loss guides backpropagation to iteratively refine both the kinematic trajectory and body scale. d Initially, the representation lacks knowledge of the person’s movements and scale (e.g., height, limb proportions), but after optimization, it typically tracks joint locations within 15 mm in 3D and 5 pixels in 2D. Biomechanical reconstruction overview. We introduce a method for biomechanically grounded movement analysis in clinical settings using a handheld smartphone. a Researchers held a smartphone (optionally with gimbal) while following a participant walking. Our system has no specific requirements regarding viewing angle, distance to subject, or therapist assistance. b Recorded smartphone video and optional wearable sensor data are stored in the cloud, and processed using PosePipe, an open-source package implementing computer vision models for person tracking and keypoint detection. c To reconstruct movement, we represent movement as a function that outputs joint angles, which–combined with body scaling parameters and evaluated through forward kinematics–generate a posed biomechanical model in 3D space. This untrained model is compared to video-extracted joint locations and optionally smartphone sensor data to compute a loss. This loss guides backpropagation to iteratively refine both the kinematic trajectory and body scale. d Initially, the representation lacks knowledge of the person’s movements and scale (e.g., height, limb proportions), but after optimization, it typically tracks joint locations within 15 mm in 3D and 5 pixels in 2D. In this study, we collected video using a Samsung Galaxy S8 or Samsung Galaxy S20 Ultra at 30 Hz and 1080 × 1920 resolution36, though most all modern smartphone video is compatible with our approach. PBL can also be paired with wearable IMUs, with all modalities closely synchronized36,38. After recording, data are uploaded to a secure cloud platform and processed using PosePipe49, an open-source software package that performs person detection, annotation, and 2D and 3D keypoint estimation. Video preprocessing Using PosePipe49, we first identified the subject of interest in each video with DeepSortYOLOv450. We then applied MeTRAbs-ACAE51 to estimate 2D and 3D virtual marker locations ("keypoints”) from the RGB video (Fig. 5b). Following the authors’ recommendation, keypoint confidence was approximated by computing the standard deviation of each 3D joint location across 10 augmented versions of each video frame. These values were mapped to confidence scores using a sigmoid function with a half-maximum at 30 mm and a width of 10 mm, yielding a confidence score c(t, j) ∈ [0, 1] for each time point t and joint j. For subsequent biomechanical fitting, we retained only frames in which the participant was fully visible, permitting short gaps ( < 1.0 s) of partial coverage. Consistent with prior work25,38, we extracted the 87 BML-MoVi keypoints from the MeTRAbs-ACAE output for biomechanical fitting. Differentiable biomechanical model Smartphone and MMMC reconstructions used a biomechanically grounded musculoskeletal model implemented in MuJoCo via LocoMuJoCo52, which is derived from an OpenSim model53. Following the notation of the skinned multi-person linear model (SMPL)54, we express the model’s forward kinematics as1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{x}}={\mathcal{M}}(\theta ,\beta ),$$\end{document}x=M(θ,β),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta \in {{\mathbb{R}}}^{40}$$\end{document}θ∈R40 denotes the joint angles that pose the model and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta \in {{\mathbb{R}}}^{8+87\times 3}$$\end{document}β∈R8+87×3 parameterizes subject-specific scaling and marker offsets. The scaling parameters include overall body size and segment-specific scale factors for the pelvis, bilateral thigh, shank, and foot segments, as well as the arms. The remaining parameters correspond to 3D offsets for each of the 87 virtual markers. The output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{x}}\in {{\mathbb{R}}}^{87\times 3}$$\end{document}x∈R87×3 represents the 3D marker locations after applying scaling, marker offsets, and forward kinematics. Forward kinematic evaluations were accelerated using MuJoCo’s GPU backend (MJX). Implicit representation We represent the kinematic trajectory of each trial as a learned implicit function fϕ, implemented as a multi-layer perceptron (MLP) that takes time as input via sinusoidal positional encoding55 and outputs time-varying joint angles θ(t). In addition to the 40 joint angles, the network also predicts three parameters describing the global orientation of the smartphone, denoted r:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f}_{\phi }:t\to (\theta ,r).$$\end{document}fϕ:t→(θ,r).Network outputs corresponding to rotational degrees of freedom (excluding pelvis translation) are passed through a \tanhtanh nonlinearity to constrain them to ( − 1, 1) and then linearly scaled to the joint limits of the biomechanical model. The resulting joint angles θ(t) are evaluated through the forward kinematic model (Eq. (1)) to obtain the 3D joint locations x(t) at each time point. Model fitting Each trial is modeled by a separate implicit function. We jointly optimize the parameters of each trial’s implicit models, ϕ0, ϕ1, …, ϕN−1, together with a single set of global body scaling and marker offset parameters β, to fit the 2D and 3D keypoints extracted from video as well as orientation data recorded by the smartphone. This strategy—jointly estimating trial-specific kinematics and session-level body parameters—has been described as bilevel optimization48. For example, a session containing 10 trials results in the simultaneous optimization of 10 implicit functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f}_{{\phi }_{1}},...,{f}_{{\phi }_{10}}$$\end{document}fϕ1,...,fϕ10 and one shared parameter vector β. During each optimization step, we evaluate all implicit functions (Eq. (2)) in parallel using batches of 300 uniformly sampled time points per trial to obtain predicted joint angles \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\theta }(t)$$\end{document}θ^(t). These joint angles, together with the shared scaling parameters, were passed through GPU-accelerated forward kinematics (Eq. (1)) to produce predicted 3D joint locations for the batch, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{{\bf{x}}}(t)={\mathcal{M}}(\widehat{\theta }(t),\beta )$$\end{document}x^(t)=M(θ^(t),β). We then computed a loss between the predicted joint locations and the corresponding MeTRAbs-ACAE observations (Fig. 5c). Reference frames We define a global reference frame n for all model outputs. Because the PBL system tracks smartphone orientation using onboard IMUs, the orientation of the camera frame c can be related to the global frame at each time point via a rotation matrix Rnc(t) ∈ SO(3). We adopt the convention of ref. 56, in which Rab denotes the orientation of frame b expressed in frame a. Losses To reduce notational clutter, we omit explicit time and trial indices in the loss definitions. MeTRAbs-ACAE provides video-only estimates of 3D keypoint locations in the camera reference frame, denoted xc51. Using the estimated camera-to-global rotation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{R}}_{nc}$$\end{document}R^nc, these keypoints are transformed into the global frame as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\bf{x}}}_{n}={\widehat{R}}_{nc}{{\bf{x}}}_{c}$$\end{document}xn=R^ncxc. We then define a 3D keypoint loss as the Euclidean distance, in centimeters, between the predicted model keypoints \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{{\bf{x}}}}_{n}$$\end{document}x^n and the transformed video keypoints:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\mathcal{L}}}_{3D}(\phi ,\beta )=\frac{1}{J}\mathop{\sum }\limits_{j\in J}c(j)g(\parallel {\widehat{{\bf{x}}}}_{n}-{\widehat{R}}_{nc}{{\bf{x}}}_{c}{\parallel }_{2})$$\end{document}L3D(ϕ,β)=1J ∑j∈Jc(j)g(∥x^n−R^ncxc∥2)where c(j) ∈ [0, 1] denotes the confidence score for keypoint j (which varies over time), and g( ⋅ ) is a Huber loss that is quadratic within 10 cm and linear beyond. This loss is computed after removing the mean translation of each 3D keypoint set. MeTRAbs-ACAE also produces 2D keypoints in the image plane, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bf{u}}\in {{\mathbb{R}}}^{87\times 2}$$\end{document}u∈R87×2. To define a corresponding 2D reprojection loss, the predicted 3D keypoints \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{{\bf{x}}}}_{n}$$\end{document}x^n are rotated into the camera frame using \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{R}}_{nc}^{-1}$$\end{document}R^nc−1 and projected through a calibrated camera model Π. The resulting pixel-space error is given by4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\mathcal{L}}}_{2D}(\phi ,\beta )=\frac{1}{J}\mathop{\sum }\limits_{j\in J}c(j)g(\parallel \Pi ({\widehat{R}}_{nc}^{-1}{\widehat{{\bf{x}}}}_{n})-{\bf{u}}{\parallel }_{2})$$\end{document}L2D(ϕ,β)=1J ∑j∈Jc(j)g(∥Π(R^nc−1x^n)−u∥2)where g( ⋅ ) is a Huber loss that is quadratic within 5 pixels and linear thereafter, reducing sensitivity to outliers. For videos collected with PBL that include phone orientation measurements, we additionally supervise the predicted camera orientation. The implicit functions are evaluated at the phone IMU sampling times to obtain a predicted orientation trajectory \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{R}}_{nc}$$\end{document}R^nc. While rotations are internally represented as rotation vectors for improved training stability, the phone’s orientation measurements are provided as quaternions. Accordingly, we convert \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{R}}_{nc}$$\end{document}R^nc to a quaternion and compute the angular difference, in degrees, between predicted (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\widehat{q}}_{nc}$$\end{document}q^nc) and measured orientations (qnc) using \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q={q}_{nc}{\widehat{q}}_{nc}^{-1}$$\end{document}q=qncq^nc−1 and then5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\mathcal{L}}}_{phone}(\theta )=\frac{180}{\pi }\cdot 2\cdot \arctan \left(\frac{\sqrt{{q}_{1}^{2}+{q}_{2}^{2}+{q}_{3}^{2}}}{| {q}_{0}| }\right)$$\end{document}Lphone(θ)=180π⋅2⋅arctanq12+q22+q32∣q0∣ Total loss and optimization These terms were combined with hyperparameters (λ1 = 1, λ2 = 1e − 1, λ3 = 1), to control their relative weights and provide an overall loss:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{L}}={\lambda }_{1}{{\mathcal{L}}}_{3D}+{\lambda }_{2}{{\mathcal{L}}}_{2D}+{\lambda }_{3}{{\mathcal{L}}}_{phone}$$\end{document}L=λ1L3D+λ2L2D+λ3Lphone We defined implicit functions and computed the loss using Equinox57 and JAX58, optimizing with the Adam optimizer for 25,000 iterations of gradient descent (Fig. 5d). Weight decay of 1e − 5 was applied to the implicit function parameters. With Optax, we set the learning rate to start at 1e − 3 and decay exponentially to 1e − 6 58,59. For MMMC25, we found random initialization of the network sufficient for optimizing implicit representations. However, for monocular videos, this often failed to converge if the subject wasn’t initially visible to the camera. To address this, we adjusted each trial’s implicit function by biasing the final layer to place the pelvis 1.5 meters from the camera and initializing Rnc outputs to the median of observed Rnc values. Run time Preprocessing a 6 second video takes 10 seconds on a server-grade GPU, 1 minute on a laptop-grade GPU and scales linearly with video length. Biomechanical fitting takes 1 minute on a server-grade GPU, <5 minutes on a laptop-grade GPU, and does not scale with video length. Dataset overview We applied our monocular fitting approach to three datasets (Fig. 1). The first is a publicly available dataset that used iPhone videos and an OMC system, but only contained able-bodied individuals. The second of these recorded clinical participants with the PBL system and also an MMMC system. The third dataset reflects our intended use case of videos recorded from participants seen in various clinics, which we used to demonstrate our method can capture clinically meaningful features of gait. Combined, these datasets contain 20.7 hours of data. OMC dataset The OMC dataset used was BML-MoVi60, a publicly available dataset of 88 healthy controls performing a diverse set of upper- and lower-body movements, including sports activities. Data were captured using iPhone 7 video at 30 Hz and optical motion capture (OMC) with an 87-marker set at 120 Hz. For the OMC data, we fit our model using only a 3D keypoint loss, as marker positions were already triangulated and cleaned. For the smartphone video, we applied both 2D and 3D keypoint losses. The smartphone recordings totaled 88 participants, 439,128 frames, and 4.1 h of video. MMMC Dataset Our MMMC dataset includes 196 participants walking and performing lower-limb functional tasks. Of these, 109 participants were recorded simultaneously using both the PBL system36 and our MMMC system25,41. This subset included 29 healthy controls, 4 pediatric inpatients, 40 lower limb prosthesis users (LLPUs), and 36 participants with neurological injury. We also recorded an additional 62 neurological inpatients, 24 LLPUs, and 1 pediatric inpatient using only the MMMC system. Control participants typically completed three recording sessions. The first session included nine walking trials, while subsequent sessions incorporated additional assessments such as the Four Square Step Test (FSST), Postural Sway Test (PST), Timed Up and Go (TUG), and tandem gait, averaging 15 recordings per session. During walking and tandem gait trials, a researcher followed the participant from behind (Fig. 5a) resulting in mostly frontal but partially oblique views. FSST was collected at a frontal view in front of the participant, and TUG was collected from a sagittal view with the camera rotating to follow the participant. Clinical participants were generally recruited from an outpatient clinic for a single walking session including the additional tasks if the participant felt comfortable. Clinical participants walked at a self selected speed using whatever assistive device they generally used (e.g., walker, cane, ankle-foot orthosis, gait belt). For clinical participants, we extracted clinical outcome scores-such as the 10 Meter Walk Test (10MWT) and Berg Balance Scale (BBS)12-from 54 participants across 95 visits. In total, the dataset includes 1,193,883 frames across 1722 trials amounting to 11.1 h of video. This study was approved by the Northwestern University Institutional Review Board (IRB# 00215263). Participants provided informed consent for their videos to be stored, processed, and de-identified videos to be shared in publication. In-Clinic dataset Our in-clinic dataset includes 74 participants recorded with the PBL system during outpatient clinic visits: 55 from a neurosurgery clinic for cervical myelopathy (CM) and 19 from a clinic for knee osteoarthritis (KOA). After their standard physician consultations, participants walked in the clinic hallway while a researcher recorded them using the PBL system. CM participants also completed the Modified Japanese Orthopedic Association (mJOA) questionnaire61,62, which scores upper and lower extremity function. Seventeen CM participants were recorded at visits both before and after surgery. The CM cohort includes 911 trials and KOA includes 110. In total, this dataset contains 589,557 frames or 5.5 h of video in clinical settings. This study was approved by the Northwestern University (IRB# 00218972) and Washington University in St. Louis Institutional Review Boards (IRB# 202111122). Participants provided informed consent for their videos to be stored, processed, and de-identified videos to be shared in publication. Kinematic accuracy evaluation To evaluate the accuracy of the smartphone-based fits compared to OMC and MMMC, we computed the Median Absolute Error, defined as the median of joint-angle errors within each trial followed by the median of these per-trial medians across trials. We used Median Absolute Error because both the within-trial and across-trial error distributions were non-Gaussian as determined by the Shapiro-Wilk test63. When aggregating errors across trials, we report the median and the normalized interquartile range (nIQR), where nIQR = IQR ⋅ 0.7413. The Median Absolute Error includes the following degrees of freedom: hips (2 × 3), knees (2 × 1), ankles (2 × 1), lumbar spine (3), neck (3), shoulders (2 × 2), and elbows (2 × 1). For comparison to OpenCap, we additionally report the Mean Absolute Error following the procedure in ref. 29. This Mean Absolute Error calculation includes the following joints: pelvis rotations (3), hips (2 × 3), knees (2 × 1), ankles (2 × 2), and lumbar (3). To compare moving (handheld) versus static camera video, we also computed monocular fits from single cameras in the MMMC dataset. Because MMMC records participants from all directions, we assessed the influence of camera viewpoint on joint-angle error by classifying viewpoint as front, back, ipsilateral, or contralateral. For static-camera and relevant PBL fits, we report the Root Translation Error (RTE), defined as the average Euclidean error between the pelvis location from the PBL reconstruction and the ground-truth OMC/MMMC reconstruction. For PBL, RTE was computed only for trials involving camera rotation but no camera translation (e.g., TUG trials). Clinical construct validity of smartphone-derived gait metrics To reduce kinematic timeseries to an interpretable score representing gait quality, we calculated the Gait Deviation Index (GDI)64. Briefly, the GDI uses a dimensionality reduction technique64,65 on cycle-aligned joint kinematics and then measures the distance in this subspace to a distribution of normative gait trials (for which we used 9008 control steps). We computed GDI, measured between 0 and 100 with 100 being normative gait, on every gait cycle for a participant and then averaged GDI over a session. Finally, we also computed double support time (DST) and cadence using a model for gait event detection that we previously developed and validated36,37. Clinical metrics must be repeatable, valid, and responsive. We assessed the repeatability of kinematics from our system by computing intra-class correlations (ICCs) of the GDI between steps taken by the same individual on the same day66. We tested construct validity of our system using t-tests to separate GDI distributions between groups. We also compared GDI with established clinical measures: the 10MWT to GDI using Pearson’s correlation coefficient, mJOA and BBS to GDI using Spearman’s rank correlation, and BBS fall-risk groups by testing for between-group differences in GDI using a t-test. For the BBS, we classify a score of ≤ 50 as a fall risk67 and ≤ 40 as high fall risk68. To evaluate the responsiveness of smartphone-based gait metrics to surgical intervention, we analyzed longitudinal pre- and post-operative changes in mJOA, cadence, DST, and GDI in a cohort of cervical myelopathy participants (n = 17) that were recorded before and after surgery. Paired changes were first assessed for statistical significance using the Wilcoxon signed-rank test. The magnitude and direction of change were then quantified using the corresponding Wilcoxon effect size, which provides a standardized measure on a [ − 1, 1] scale69. Ninety-five percent confidence intervals for the effect sizes were estimated via bootstrapping. In addition, changes in raw knee kinematics before and after surgery were assessed using paired SPM1D t-tests39,40. Supplementary information Supplementary information Supplementary information