UCOPhyRehab++: A multi-modal and multi-view dataset for human rehabilitation analysis Authors: Rafael Aguilar-Ortega, Jorge Zafra-Palma, Rafael Muñoz-Salinas, Manuel J Marin-Jimenez Published: 2026 ABSTRACT AbstractThe rehabilitation of patients with musculoskeletal disorders is usually associated with the performance of prescribed exercises at home. Performing these exercises without medical supervision may lead to incorrect execution, resulting in secondary injuries or slower recovery rates for these patients. For this reason, research into assisted rehabilitation methodologies for patients of this type has been one of the most studied fields in recent years. The use of computer vision techniques has rapidly increased in recent literature. However, there is a significant lack of available data for training machine learning models or for testing these systems. In this paper, we extend our previous work, UCOPhyRehab (University of COrdoba Physical Rehabilitation), by adding multiple modalities to the original data, incorporating demographic metadata, and including performance scores assigned by an expert physical therapist. Our validation experiments demonstrate that this new release complements the original UCOPhyRehab data and enables new research directions, such as multi-modal fusion (e.g., combining silhouettes, optical flow, and semantic segmentation) and multi-view fusion across the five camera viewpoints, to improve the robustness and accuracy of rehabilitation-assistance methods. Background & Summary Physical rehabilitation is essential for the recovery and long-term well-being of individuals with musculoskeletal injuries, neurological disorders, or post-surgical conditions. The success of rehabilitation programs depends not only on the prescribed protocol but also on the correct execution of each exercise. Traditionally, the assessment of exercise quality has relied on direct supervision by healthcare professionals1, specialized motion-capture (MoCap) systems2, such as OptiTrack, or wearable devices and specific hardware3, such as Inertial Measurement Units (IMUs). While accurate, these systems are expensive, require controlled laboratory settings, are invasive, and are not easily scalable to daily clinical practice or home-based rehabilitation scenarios. Recent advances in computer vision4 have opened the possibility of monitoring rehabilitation exercises using low-cost RGB cameras, making the development of automated systems more feasible. Most of the previous research focused on the Range of Motion (ROM) of the patients5,6, depth information7 with specific devices, Inertial Measurement Units (IMU)8, or motion-capture (MoCap)9. However, research progress in this area has been slowed by the lack of openly available datasets that combine multi-modal information with clinically meaningful annotations. There are several general-purpose datasets for human pose estimation or action recognition, such as Human3.6M10 and AMASS11. For health-related works, most of the works rely on the Gait Analysis, like Health&Gait dataset12, where gait analysis information and general demographic metadata were included, or Scoliosis1K13, where the authors tried to detect scoliosis using gait analysis as input. As we can see, there is a lack of general rehabilitation exercise datasets with expert-based quality assessments or demographic metadata, which are crucial for healthcare-related applications. To fill this gap, we present UCOPhyRehab++, an extended version of the UCOPhyRehab dataset originally introduced in14. This new release provides multi-modal visual representations derived from RGB recordings (silhouettes, optical flow, and semantic segmentation), together with detailed demographic metadata (sex and age). A major novelty is the inclusion of clinical annotations. Each repetition of each exercise has been independently assessed by a PhD physiotherapist, who rated the quality of execution based on the range of motion of the involved joints and the overall body posture. By combining multi-modal visual data with demographic and clinical annotations, UCOPhyRehab++ provides a unique benchmark for developing computer vision algorithms in rehabilitation. The dataset enables a wide variety of research directions, including the automatic evaluation of exercise quality, demographic-aware analysis of movement patterns, and multi-modal fusion strategies that integrate clinical knowledge. All data and metadata are publicly available through Zenodo15, ensuring reproducibility and accessibility for the scientific community. Method Experimental design The previous version of this dataset, proposed in14, consisted of 2160 video sequences recorded from five different camera angles with a resolution of 1280 × 720 pixels. It included 27 healthy subjects (20 males and 7 females; mean age 35.52 ± 10.58) who performed 16 different physical rehabilitation exercises for the lower and upper limbs, and was recorded from June to July 2022. The dataset was recorded in the Artificial Vision Applications lab at the University of Córdoba during the development of the RehApp project. The study protocol was approved by the Ethics Committee of the Reina Sofía University Hospital (Córdoba, Spain), with reference number 5071, under the HIP-CI project, version 1, entitled “ReHApp: Analysis by vision of postoperative evolution in surgical interventions”. All participants provided written informed consent for data collection and for the public release of anonymised derived data. The dataset shared in this work includes only non-identifiable representations, namely optical flow, silhouettes, and semantic segmentation masks, which do not allow participants re-identification. The subjects were recruited from the city of Córdoba, and the inclusion criteria were age (older than 18 years) and the ability to read, understand, and sign the corresponding informed consent to be recorded. In addition, subjects were informed about future methods for distributing data for scientific purposes. Subjects were recruited regardless of physical condition to ensure greater variability. The aim of the recording of this dataset was to compare the performance of state-of-the-art pose estimation models available at the time of the original study when applied to a specific use case: postoperative physical rehabilitation. This was addressed in the mentioned study14, where the authors evaluated how these models behave in the context of rehabilitation exercises, with a focus on their accuracy and reliability for assisting patients in their recovery process. Data acquisition In this new release, each participant is now associated with demographic metadata (sex and age), and every repetition of each exercise has been independently evaluated and scored by two PhD physiotherapist from the University Hospital of Jaén (Spain) and from the Federal University of São Carlos (Brasil), providing expert-based quality labels that can be used for automatic assessment of rehabilitation performance. The scoring process followed a structured 5-point ordinal scale based on three objective biomechanical pillars: Range of Motion (ROM), Postural Control, and Execution Velocity. The therapist evaluated whether the subject executed the full anatomical trajectory (e.g., reaching the vertical plane in shoulder flexion), maintained a stable spinal alignment without compensatory movements, and adhered to an optimal angular velocity (ω ≈ 45°/s) indicative of proper motor control. The complete grading criteria are detailed in Table 1. Table 1Clinical Assessment Rubric for Movement Quality.ScoreClassificationBiomechanical Criteria5OptimalFull ROM achieved according to individual baseline. Rigid postural stability with no compensatory movements. Target velocity (approx. 45 deg/s) throughout the cycle.4FunctionalFull or near-complete ROM. Minor postural sway without compromising the primary joint action. Consistent velocity with negligible fluctuations.3Sub-optimalIncomplete ROM. Moderate compensations (e.g., trunk leaning or spinal detachment from support). Irregular velocity or lack of fluid control.2DeficientSignificant ROM limitation. Poor postural control; movement triggers global instability in the kinetic chain. Inadequate velocity (reliance on momentum).1MinimalTrace movement or null ROM. Inability to maintain baseline posture. Non-functional execution preventing kinematic evaluation. Clinical Assessment Rubric for Movement Quality. To assess the reliability of the clinical annotations, two PhD physiotherapists independently scored the complete dataset following the rubric defined in Table 1. Both raters performed their annotations independently and were blinded to subject demographics and to each other’s scores. Inter-rater agreement was evaluated using the intraclass correlation coefficient (ICC)16 and Cohen’s weighted kappa17 The ICC(C,1) was 0.41(95%CI: 0.38 − 0.45), and Cohen’s quadratic weighted kappa was 0.41, both indicating moderate agreement according to standard interpretation criteria16. Exact agreement between raters was observed in 64.1% of repetitions, and 97.0% of annotations differed by at most one score point on the 5-point ordinal scale. The moderate ICC value is consistent with the inherent subjectivity of clinical movement assessment, where fine-grained distinctions between adjacent score levels (e.g., 4 and 5) are particularly challenging even for trained specialists. The high proportion of annotations within one point of agreement (97.0%) reflects a solid practical consistency between raters, supporting the reliability of the dataset for training and evaluating machine learning models in rehabilitation contexts. Importantly, instead of generating separate video clips, the start and end frames of each repetition have been annotated in the metadata files, thus allowing researchers to isolate and analyze individual repetitions without altering the original recordings. The repetitions of the exercises have a minimum length of 15 frames (e.g., circular pendulum), a maximum length of 675 frames, and an average length of 151 frames. In addition, several new data modalities have been derived from the original RGB recordings, inspired by recent datasets such as Health&Gait12: Silhouettes, Optical Flow and Semantic Segmentation. Together with the previously available RGB videos and 3D ground-truth joint positions, these additions make the dataset a rich multi-modal resource for studying rehabilitation exercises. Data processing To ensure a reliable performance evaluation, we split each sequence into multiple repetitions of the same exercise. For that purpose, we monitored one key joint per exercise to identify its maximum and minimum ranges of motion within each sequence. A Fourier filter was applied to remove high-frequency noise, using an adaptive cutoff set to the dominant frequency of each signal (i.e., the frequency with the highest power in the spectrum), and peaks and valleys of the filtered signal were used to determine the start and end of each repetition. An example is shown in Figure 1. These split subsequences were then linked to the corresponding physiotherapist score in the metadata files. This strategy ensures that complete and isolated repetitions can be accurately retrieved without explicitly segmenting the video sequences. Filtered plot using Fourier filtering for the Bending the knee with support while sitting exercise, highlighting the peaks and valleys used to clip repetitions from the videos in the dataset, four of which are shown in this example. (The individual shown in the images consented to their publication). From the original RGB sequences, three additional visual modalities were derived to enrich the dataset and facilitate multi-modal analyses. Specifically, we computed silhouettes using instance segmentation (YOLO11x-seg18) to capture the body outline while discarding background information; optical flow using both traditional (TV-L119) and learning-based (GMFlow20) methods to represent dense motion fields between consecutive frames; and semantic segmentation through DensePose-based models, assigning a body-part label to each pixel of the subject and thus enabling fine-grained motion and posture analysis. All the modalities can be visualized in the Figure 2. Examples of the new modalities incorporated into the dataset. Each row corresponds to a different rehabilitation exercise, while each column shows one of the generated modalities: (a) RGB frame, (b) binary silhouette, (c) semantic segmentation, (d) optical flow computed with TV-L1, and (e) optical flow computed with GMFlow. It should be noted that the bands and markers visible on the participants’ limbs in the RGB modality are passive markers belonging to the OptiTrack motion-capture (MoCap) system. (The individual shown in the images consented to their publication). All processing steps were designed to ensure temporal alignment across modalities, so that each frame in the RGB sequences has its corresponding silhouette, optical flow, and semantic segmentation representation. The resulting data are stored in standard image formats to guarantee usability and easy integration with deep learning frameworks, with a complete resolution of 1280 × 720 pixels. Due to storage constraints, the sequences have been compressed into MP4 format (preserving the original quality), but the corresponding source code is provided in the GitHub repository for decomposing them into separate frames and replicating the conditions of the experiments described in this article. Silhouettes Body silhouettes were extracted using an instance segmentation model (YOLO11x-seg pre-trained on COCO dataset21). This approach produces binary masks that capture the subject’s body outline while removing background information. Optical flow Motion information between consecutive frames was estimated using two complementary approaches: the classical TV-L1 algorithm19 and the deep learning-based GMFlow method20. To facilitate visualization and reduce storage requirements, the optical flow fields were encoded as color images (HSV for TV-L1 and Middlebury color code for GMFlow). Also, for the GMFlow modality, binary masks were applied to remove noisy backgrounds. Semantic segmentation Pixel-level body-part segmentation was obtained using DensePose model22, which assigns each pixel of the subject to one of several predefined body regions. For exercises performed on a treatment table, participants were shown horizontally in the RGB frames, which caused DensePose to struggle to extract accurate silhouettes. To overcome this limitation, we applied a preprocessing step in which each affected frame was temporarily rotated to make the person appear upright. Semantic segmentation was then computed on the rotated image, and the resulting mask was subsequently rotated back to its original orientation. This procedure improved segmentation reliability while preserving the original spatial configuration of the data as it’s shown in the Fig. 3. This rotation-based procedure was applied to every frame of the sequences where the subject performed exercises in a supine position (exercises 03, 04, 07 and 08), as these are the only exercises in the dataset in which the participant lies horizontally on the treatment table. This representation provides fine-grained motion and posture information that complements pose-based descriptors. The results are stored as color-coded images, where each color corresponds to a specific body part. Correspondences between specific body parts and colors are shown in Table 2. Effect of frame rotation on semantic segmentation quality for supine exercises. Each row shows a representative frame from a different exercise sequence performed on a treatment table (exercises 03-08), comparing the segmentation obtained with the rotation preprocessing step (left column) against the result without it (right column). Since DensePose was trained predominantly on upright subjects, applying the rotation prior to segmentation and reverting it afterward yields a more complete and accurate body-part labeling, particularly in the assignment of limb regions. Without rotation, several body parts are either missing or incorrectly labeled, whereas the rotated version recovers a coherent full-body segmentation. Color coding follows Table 2. Table 2Color distribution for the specific body parts in the semantic segmentation modality.Part of the bodyColorHex CodeTorso#ff0000Right hand#0000ffLeft hand#00ff00Right foot#800000Left foot#ffff00Upper leg right#808000Upper leg left#008080Lower leg right#800080Lower leg left#808080Upper arm right#ff0080Upper arm left#ff8000Lower arm right#80ff80Lower arm left#ff80ffHead#ffffff Color distribution for the specific body parts in the semantic segmentation modality. This multi-modal extension broadens the scope of the dataset beyond pure RGB and pose information. By providing aligned visual representations that capture shape, motion, and body-part semantics, the dataset supports research in multi-modal learning. Importantly, all modalities and annotations are synchronized at the level of individual repetitions, enabling fine-grained analyses that were not feasible in the original version. Data Records All data are publicly available through Zenodo (15), following the same open-access policy as similar resources such as Health&Gait12. The dataset is organised by subject and exercise, with separate directories for each visual modality (silhouettes, OF GMFlow, OF TV-L1, and semantic segmentation). Each modality is provided in MP4 video format, facilitating direct use in computer vision frameworks. The visual modalities are provided as MP4 files encoded with H.264 (MPEG-4 AVC) at a resolution of 1280 × 720 pixels. While these files can be used directly with modern data loaders, we recommend frame extraction for typical deep-learning pipelines to ensure random access efficiency and to avoid potential inter-frame interpolation artifacts during training. To facilitate this, the publicly available repository includes scripts to systematically decompose the sequences into individual frames while preserving original quality. Due to privacy concerns, RGB data will not be available in the Zenodo repository. This modality is only available upon request for research purposes. In addition to the image data, a metadata file named ucophyrehab2_data.jsonl is provided. Each line in this JSONL file corresponds to one exercise performed by a specific subject. The structure of the entries is as follows: subject_id: anonymised subject identifier (e.g., "00"). subject_id: anonymised subject identifier (e.g., "00"). exercise_id: identifier of the performed exercise (e.g., "01"). See Table 4 for the full list. exercise_id: identifier of the performed exercise (e.g., "01"). See Table 4 for the full list. sex: sex of the participant ("M" or "F"). sex: sex of the participant ("M" or "F"). age: participant’s age in years (e.g., 59). age: participant’s age in years (e.g., 59). age_range: age group to which the participant belongs (e.g., "51+"). See Table 3 for the full list. age_range: age group to which the participant belongs (e.g., "51+"). See Table 3 for the full list. scores: list of annotated repetitions, where each repetition is defined by: init_frame: initial frame index of the repetition.final_frame: final frame index of the repetition.score: list of quality scores assigned independently by PhD physiotherapists, each rating the execution quality on a 5-point ordinal scale (1–5), where higher scores indicate better execution. Each value corresponds to one rater, in the order in which the annotations were incorporated. scores: list of annotated repetitions, where each repetition is defined by: init_frame: initial frame index of the repetition.final_frame: final frame index of the repetition.score: list of quality scores assigned independently by PhD physiotherapists, each rating the execution quality on a 5-point ordinal scale (1–5), where higher scores indicate better execution. Each value corresponds to one rater, in the order in which the annotations were incorporated. init_frame: initial frame index of the repetition. init_frame: initial frame index of the repetition. final_frame: final frame index of the repetition. final_frame: final frame index of the repetition. score: list of quality scores assigned independently by PhD physiotherapists, each rating the execution quality on a 5-point ordinal scale (1–5), where higher scores indicate better execution. Each value corresponds to one rater, in the order in which the annotations were incorporated. score: list of quality scores assigned independently by PhD physiotherapists, each rating the execution quality on a 5-point ordinal scale (1–5), where higher scores indicate better execution. Each value corresponds to one rater, in the order in which the annotations were incorporated. Table 4List of exercises included in the original UCOPhyRehab dataset14.#Exercise nameSideUpper / Lower Body01Bending the knee without support while sittingLeftLower02Bending the knee with support while sitting03Lift the extended leg04Bending the knee with bed support05Bending the knee without support while sittingRight06Bending the knee with support while sitting07Lift the extended leg08Bending the knee with bed support09Shoulder flexionLeftUpper10Horizontal weighted openings11External rotation of shoulders with elastic band12Circular pendulum13Shoulder flexionRight14Horizontal weighted openings15External rotation of shoulders with elastic band16Circular pendulumThey are classified by position, side and upper/lower body. List of exercises included in the original UCOPhyRehab dataset14. They are classified by position, side and upper/lower body. Table 3Age groups by range included in the dataset.Age range (years)Number of sequences20–3015631–409241–5013351+30 Age groups by range included in the dataset. This structure allows researchers to access both the demographic metadata and the clinical annotations at the level of individual repetitions without altering the original video sequences. By combining the information in the JSONL file with the corresponding visual modalities, users can retrieve any repetition along with its multi-modal representations and expert evaluation. In total, the dataset contains 27 subjects performing 16 different rehabilitation exercises, resulting in 2160 video sequences recorded from five camera viewpoints. The metadata file includes 411 exercise-level entries, each containing the participant’s demographic information and a list of repetitions with their associated frames and scores. A total of 21 sequences have been dismissed due to errors in the original 3D data acquisition. Regarding annotation effort, the scoring process was time-intensive, requiring an average of 33 hours per specialist to complete the full annotation of the dataset. Figure 4 shows the distribution of quality scores across all annotated repetitions, where scores 4 and 5 account for the majority of annotations. Distribution of expert-assigned quality scores across all annotated repetitions (1,992 sequences). Scores follow the 5-point ordinal scale described in Table 1. Regarding the distribution of scores across exercises, the mean score (averaged across both raters) ranges from 4.28 ± 0.53 (Ex. 12, circular pendulum) to 4.58 ± 0.46 (Ex. 01, bending the knee without support while sitting), indicating that scores are consistently high across all exercises with no marked differences between them. Lower-limb exercises (Ex. 01–08) show a slightly higher mean (4.45 ± 0.52) than upper-limb exercises (Ex. 09–16, 4.36 ± 0.52), though the difference is negligible. This uniformity is consistent with the fact that subjects are healthy individuals performing prescribed exercises under controlled conditions. Another file has been released in order to integrate the old ground-truth of the previous release with the new provided data, dataset_3d_with_angles.json. This JSON file stores both sequence-level metadata and frame-level ground-truth annotations. For each exercise repetition, it specifies the folder in which the sequence is located, the exercise identifier, the acquisition position (e.g., seated), the exercised side (left/right), the body region (e.g., lower limb), the total number of frames, the number of frames with valid angle annotations, and summary statistics of the joint angle over the sequence (mean, minimum, and maximum values). The frames field then contains a list of all annotated frames, starting from frame 5, since the first five frames of every sequence were discarded because the ground-truth angle was not reliably captured at the beginning of the movement. Each frame entry includes the frame index (id) and the three-dimensional coordinates (x, y, z) of the relevant anatomical landmarks for the joint under analysis (e.g., left hip, knee, and ankle for lower-limb exercises), together with the corresponding ground-truth joint angle in degrees, which is later used as the target for training and evaluation. Technical Validation To validate the quality of the new data, we conducted two complementary sets of experiments. First, we trained models to predict the 3D angle formed by the main joint involved in each exercise, using the MoCap-derived ground-truth as the regression target. This objective kinematic parameter provides a deterministic and measurable evaluation criterion, allowing a rigorous comparison across modalities and viewpoints. Second, we trained models to predict the expert quality scores assigned by the physiotherapists, directly addressing the clinical utility of the new annotations. While the angle prediction task benefits from an exact and unambiguous ground-truth, the score prediction task is inherently bounded by the inter-rater variability of the annotations themselves, as reflected by the moderate ICC and weighted kappa values reported in the Data Acquisition section, which sets a practical ceiling on the performance achievable by any model. Angle prediction The predicted angle is always the angle formed by the three joints involved in the specific exercise, as shown in Figure 5. The main reason to choose the angle as a technical validator is that it represents a fully objective kinematic parameter directly derived from the 3D ground-truth. This ensures a deterministic and measurable evaluation of how well each new modality captures the physical movement. The goal of these experiments is therefore to demonstrate the technical feasibility and potential utility of the additional modalities, rather than to match or surpass the current state of the art in pose estimation or movement analysis. Evolution of the measured angle during a specific repetition of the rehabilitation exercise (Subject 25, Exercise 1, Camera cam0). The top row presents four RGB snapshots captured at frames with IDs 13, 29, 88, and 110. The main plot shows the angle value (in degrees) formed by the hip, knee and ankle joints over the sequence of frames. Each red marker on the curve corresponds to one of the sample frames above, allowing for direct visualization of the kinematic state at that specific time point. (The individual shown in the images consented to their publication). We trained a model for each specific pair of sequences using the right and left variants for the same exercise (see Table 4). For example, for the exercise “Bending the knee without support while sitting” we trained a model using the sequences “01” and “05” (see Table 4 for reference). For each of these experiments, we compared the performance of the modalities across pairs of exercise sequences and across camera viewpoints. Additionally, we trained a model for each exercise pair using all camera viewpoints. The regression model used for the technical validation is based on a ResNet-1823 architecture, implemented in PyTorch 2.9. The backbone was modified in its first layer to accept the specific input channels of each modality, and the final fully connected layer was replaced with a regression head consisting of two linear layers with a ReLU activation and a hidden dimension of 512. The model was trained using a Mean Absolute Error (MAE) loss function and the Adam optimizer with a learning rate of 0.0001 for 30 epochs using a four-fold cross-validation scheme, with the specific partitioning configurations detailed in Table 5. Images were resized to 224 × 224 pixels and normalized before being fed into the network. A batch size of 32 was used across all experiments to ensure stable convergence. The choice of k = 4 for the cross-validation scheme was determined by the total number of participants (27 subjects) and the objective of maintaining a robust 80/10/10 split (train/validation/test) within each fold. This configuration allows for 21 subjects to be used for training, providing a large enough sample for the model to learn movement patterns, while reserving 3 subjects for validation and 3 for independent testing. This ensures that the results are not biased by a specific subject’s performance and that the evaluation is conducted on completely unseen data for each fold. The performance was evaluated using the Mean Absolute Error (MAE) in degrees. Table 5Subject distribution for the four cross-validation splits used in our experiments.SplitSet#SubjMFMean Age ± SD1Train2116536.7 ± 9.9Val31239.0 ± 4.6Test31234.3 ± 7.62Train2116536.1 ± 8.8Val32135.0 ± 12.1Test32133.7 ± 5.33Train2116535.9 ± 9.4Val31236.3 ± 12.6Test32128.7 ± 5.44Train2116535.5 ± 9.3Val32131.3 ± 11.2Test31236.3 ± 7.4Each split follows an 80/10/10 partition (train/validation/test), ensuring no subject overlap across sets and maintaining balanced distributions of sex and age. The table reports, for every split and partition, the number of subjects, sex composition, and age statistics in each group. Subject distribution for the four cross-validation splits used in our experiments. Each split follows an 80/10/10 partition (train/validation/test), ensuring no subject overlap across sets and maintaining balanced distributions of sex and age. The table reports, for every split and partition, the number of subjects, sex composition, and age statistics in each group. To further exploit the sequential nature of rehabilitation movements and evaluate multi-modal fusion strategies, we additionally trained a CNN+LSTM model that processes temporal windows of T = 32 frames with a stride of 8 frames, producing a dense angle prediction for each frame in the window. The model consists of two independent ResNet-18 encoders, one for the silhouette modality (1-channel input) and one for the GMFlow optical flow modality (3-channel input), both trained from scratch. Each encoder processes the T frames in parallel and produces a 512-dimensional feature vector per frame, which are then concatenated into a 1024-dimensional representation, following a mid-fusion strategy. This combined sequence is fed into a 2-layer LSTM with a hidden size of 256 units and a dropout of 10% between layers, followed by a fully connected regression head with architecture Linear(256 → 256) → ReLU → Linear(256 → 1), yielding one angle prediction per frame. The model was trained using MSE loss and the same Adam optimizer and cross-validation scheme described above. Quality score prediction For quality score prediction, we trained two model variants sharing the same two-stage architecture. The spatial stage consists of a ResNet-18 encoder trained from scratch, with the first convolutional layer adapted to single-channel input and the fully connected layer replaced by a linear projection (512 → 256, ReLU), producing a spatial feature vector per frame. The temporal stage consists of a 2-layer LSTM with a hidden size of 128 units and a dropout of 0.3 between layers, which processes a sequence of T = 32 frames uniformly sampled from each repetition segment, taking the output of the last temporal step as the segment-level representation. The two variants differ only in their prediction head: the regression variant produces a continuous scalar via a Linear(128, 1) layer, while the classification variant outputs logits over five ordinal classes via a Linear(128, 5) layer. The ground truth for both variants is derived from the mean of the two rater scores; for classification, the target class is obtained as round(s¯)∈{1,…,5}. Both variants were trained using the Adam optimizer (lr = 1 × 10−4, weight decay = 0.05) with a batch size of 8, gradient clipping (norm 1.0), and early stopping with a patience of 3 validation checks. The regression variant uses MSE loss, while the classification variant uses cross-entropy with label smoothing ε = 0.1 to reduce overconfidence given the inherent inter-rater variability of the annotations. Validation results Silhouettes The silhouettes modality consistently provided robust and relatively low errors for the 3D angle prediction task (see Table 6 top-left), capturing the body outline while discarding background information. The lowest overall error across all experiments was achieved for exercise pair Ex. 01-05 with a MAE of 3.57 ± 0.89 in the cam0 view. Moreover, the cam0 viewpoint delivered the best performance for three out of the four tested lower-limb exercise pairs (Ex. 01-05, Ex. 03-07, and Ex. 04-08), while Ex. 02-06 obtained its best result in cam4. This pattern suggests that a frontal perspective is highly effective when relying solely on body-shape information, although slightly oblique viewpoints can be advantageous for certain movement patterns. Table 6Results for 3D angle prediction over all modalities.SilhouettesMAE ↓Ex. 01-05Ex. 02-06Ex. 03-07Ex. 04-08cam03.57 ± 0.894.41 ± 0.813.86 ± 1.403.80 ± 0.35cam13.77 ± 0.564.74 ± 0.984.69 ± 0.934.53 ± 0.60cam23.66 ± 0.485.09 ± 1.173.98 ± 1.243.92 ± 0.16cam34.27 ± 0.674.95 ± 0.964.21 ± 1.264.11 ± 0.26cam44.23 ± 0.654.34 ± 0.494.22 ± 1.164.32 ± 0.81Optical flow (GMFlow)cam08.55 ± 1.126.46 ± 0.754.72 ± 0.9215.98 ± 1.87cam19.69 ± 0.78.47 ± 0.185.35 ± 0.6417.75 ± 2.05cam210.35 ± 0.529.61 ± 0.54.63 ± 0.7618.93 ± 2.24cam312.39 ± 1.6511.62 ± 0.95.6 ± 0.6821.38 ± 2.19cam413.32 ± 0.5712.25 ± 0.954.92 ± 1.2421.11 ± 5.75Optical flow (TV-L1)cam05.00 ± 0.884.32 ± 0.574.69 ± 0.707.32 ± 0.79cam16.24 ± 0.576.37 ± 0.696.19 ± 1.228.18 ± 1.91cam25.33 ± 0.465.75 ± 1.034.88 ± 0.666.92 ± 1.57cam35.34 ± 0.926.29 ± 0.886.15 ± 0.617.82 ± 1.68cam46.19 ± 1.135.07 ± 0.224.95 ± 0.785.53 ± 0.26Semantic segmentationcam04.33 ± 0.723.72 ± 1.214.56 ± 1.225.38 ± 1.15cam15.06 ± 0.525.78 ± 0.874.99 ± 0.815.47 ± 0.41cam23.94 ± 0.583.99 ± 0.874.57 ± 0.374.57 ± 0.94cam34.27 ± 0.245.03 ± 0.765.75 ± 0.545.54 ± 0.83cam44.38 ± 0.664.91 ± 0.483.68 ± 0.765.03 ± 0.81The Mean Absolute Error (MAE) in degrees (↓) is reported for the angle of the main joint involved in each exercise. Columns correspond to pairs of exercises (left and right variants, as listed in Table 4), and rows correspond to different camera viewpoints (cam0 to cam4). The best result for each exercise pair within each modality is highlighted in bold. Results for 3D angle prediction over all modalities. The Mean Absolute Error (MAE) in degrees (↓) is reported for the angle of the main joint involved in each exercise. Columns correspond to pairs of exercises (left and right variants, as listed in Table 4), and rows correspond to different camera viewpoints (cam0 to cam4). The best result for each exercise pair within each modality is highlighted in bold. Optical Flow The results for the optical flow (GMFlow and TV-L1) modalities (see Table 6 top-right and bottom-left), which focus on dense motion fields between consecutive frames, revealed different behaviors. GMFlow produced highly variable performance: while a low MAE of 4.63 ± 0.76 was obtained for Ex. 03-07 in cam2, the errors for Ex. 01-05, Ex. 02-06, and Ex. 04-08 were substantially higher, ranging approximately from 6.46° to 21.38°. In contrast, the TV-L1 optical flow yielded considerably more stable and accurate predictions, with MAE values typically between 4° and 8°. The best results for each exercise pair using TV-L1 were 5.00 ± 0.88 (Ex. 01-05, cam0), 4.32 ± 0.57 (Ex. 02-06, cam0), 4.69 ± 0.70 (Ex. 03-07, cam0), and 5.53 ± 0.26 (Ex. 04-08, cam4). Overall, these findings indicate that, although pure motion cues are less informative than silhouette information for precise angle regression, a more traditional TV-L1 formulation produces significantly more reliable features than GMFlow in this setting. This may be due to the fact that TV-L1, with its simpler motion model, captures more stable and coherent motion patterns compared to GMFlow. Nonetheless, for completeness, we also present GMFlow results for comparison. Semantic Segmentation The semantic segmentation modality (see Table 6 bottom-right), which provides pixel-level body-part information, showed competitive performance, often comparable to that of silhouettes. Several combinations of exercise pair and viewpoint yielded MAE values below 4°, such as 3.94 ± 0.58 for Ex. 01-05 in cam2, 3.72 ± 1.21 for Ex. 02-06 in cam0, and 3.68 ± 0.76 for Ex. 03-07 in cam4. The inclusion of fine-grained, body-part-specific information appears to complement pose-based descriptors effectively, providing robust angle estimation across multiple viewpoints rather than favoring a single dominant camera. All viewpoints Finally, we assessed the effect of training the models using all camera viewpoints simultaneously to evaluate their generalization capability and robustness. As summarized in Table 7, using data from all viewpoints often led to a performance that was either better than or comparable to the best single-view result. In particular, the silhouettes and semantic segmentation modalities maintained solid performance across most exercise pairs when trained with multi-view data. This outcome validates the utility of the dataset’s multi-view nature, confirming that combining different perspectives provides a more comprehensive spatial understanding, which is crucial for building robust models for 3D angle regression in physical rehabilitation. In summary, the technical validation confirms the practical utility of the new visual modalities and the clinical annotations in UCOPhyRehab++ for regression tasks, positioning silhouettes, and semantic segmentation as key data sources for future algorithm development. Table 7Results for 3D angle prediction over each modality and all viewpoints.All viewpoints (MAE↓)Ex. 01-05Ex. 02-06Ex. 03-07Ex. 04-08AVGSilhouettes3.71±0.47_3.98 ± 0.843.51 ± 0.984.05±0.18_3.81OF GMFlow11.32 ± 0.879.79 ± 0.754.8 ± 0.9318.21 ± 1.4911.03OF TV-L15.23 ± 0.454.63 ± 0.485.11 ± 0.775.64 ± 0.545.15Semantic segmentation3.89 ± 0.464.18 ± 0.764.22 ± 0.774.25 ± 0.364.14CNN+LSTM (Sil + OF GMFlow)3.39 ± 0.173.99±0.59_4.14±0.97_3.79 ± 0.223.83_Results from14 — global AVG across all cameras and lower-body exercisesAlphaPose—6.75MediaPipe—9.36HMR—13.84VPose3D—9.11KAPAO—7.84Strided—9.88The average Mean Absolute Error (MAE) in degrees (↓) for each modality, including all viewpoints, is reported for the angle of the main joint involved in each exercise. Best results for each exercise group are highlighted in bold. Results from14 are included for reference, reporting only the global model average across all cameras and lower-body exercises. Best results are highlighted in bold, and second-best results are underlined. Results for 3D angle prediction over each modality and all viewpoints. The average Mean Absolute Error (MAE) in degrees (↓) for each modality, including all viewpoints, is reported for the angle of the main joint involved in each exercise. Best results for each exercise group are highlighted in bold. Results from14 are included for reference, reporting only the global model average across all cameras and lower-body exercises. Best results are highlighted in bold, and second-best results are underlined. CNN+LSTM multimodal The results for the CNN+LSTM model trained with all camera viewpoints are reported in Table 7. This model, which combines silhouette and GMFlow optical flow modalities through a temporal architecture, achieved competitive performance across all exercise pairs, with average MAE values of 3.39 ± 0.17, 3.99 ± 0.59, 4.18 ± 1.05, and 3.79 ± 0.22 degrees for exercise pairs Ex. 01-05, Ex. 02-06, Ex. 03-07, and Ex. 04-08, respectively. These results are broadly comparable to those of the single-modality baselines in the same table. It should be noted that the CNN+LSTM model was trained for the same number of epochs as the single-modality baselines, despite having a significantly larger number of parameters, which likely limits its performance in this setting. A more thorough exploration of temporal and multi-modal architectures, including extended training schedules and hyperparameter tuning, is left as future work, as the primary goal of this paper is to introduce and validate the dataset rather than to optimize model performance. Quality score prediction Table 8 reports the results for both the regression and classification variants across the four exercise groups, averaged over the four cross-validation splits. The regression variant achieved a mean MAE of 0.393 across all exercise groups, which is comparable to the inter-rater disagreement of 0.36 observed between the two physiotherapists, suggesting that the model is operating near the practical performance ceiling imposed by the annotation variability itself. The relaxed accuracy within one score point (i.e., the percentage of predictions deviating by at most one ordinal level from the ground truth) reached approximately 99.4% and 99.7% for the regression and classification variants, respectively, indicating that in virtually all cases the predicted score does not deviate from the ground truth by more than one ordinal level – a clinically meaningful result. The Spearman correlation and weighted kappa values reflect moderate but non-trivial agreement with the expert annotations, consistent with the difficulty of the task and the limited sample size. The high variance across splits (e.g., Spearman ρ ranging from −0.05 to 0.68 depending on the split and exercise group) reflects the small number of test subjects per fold and the heterogeneity between subgroups. Both variants show complementary strengths: the regression model captures fine-grained score differences useful for quantifying movement quality on a continuous scale, while the classification model produces discrete labels directly interpretable in a clinical context. Table 8Quality score prediction results for the regression and classification variants, averaged over four cross-validation splits.Score prediction (Regression)Ex. 01+05Ex. 02+06Ex. 03+07Ex. 04+08MAE↓0.3560.3840.4110.421RMSE↓~0.432~0.463~0.501~0.506Rel. Acc ± 1↑99.6%99.3%98.6%100.0%Spearmanρ↑0.2910.3650.3030.286κw↑0.1710.2980.2560.169MeanMAE: 0.393 Rel. Acc: ~ 99.4% ρ: 0.311 κw: 0.223Score prediction (Classification)Top-1 Acc↑56.8%52.9%61.9%66.1%Rel. Acc ± 1↑99.4%99.6%100.0%100.0%Spearmanρ↑0.1450.1740.2890.248κw↑0.1090.1040.2240.245MeanTop-1: 59.4% Rel. Acc: ~ 99.7% ρ: 0.214 κw: 0.171Relaxed Acc ± 1 denotes the percentage of predictions within one score point of the ground truth. Spearman ρ and weighted kappa κw measure rank correlation and ordinal agreement with the expert annotations, respectively. Quality score prediction results for the regression and classification variants, averaged over four cross-validation splits. Relaxed Acc ± 1 denotes the percentage of predictions within one score point of the ground truth. Spearman ρ and weighted kappa κw measure rank correlation and ordinal agreement with the expert annotations, respectively. To address the question of statistical significance, we applied the Wilcoxon signed-rank test to compare MAE values across modalities over the four cross-validation splits. However, with only n = 4 folds, the minimum achievable p-value for this test is 0.125, which precludes any statistically significant result regardless of the observed differences. This is an inherent limitation of the experimental design, imposed by the reduced number of subjects (27) and the constraint of maintaining subject-independent partitions. The observed trends should therefore be interpreted as indicative rather than conclusive, and future work with larger cohorts will be necessary to establish statistical significance. Figure 6 provides a compact graphical overview of the angle prediction results under both single-viewpoint and all-viewpoints training configurations, including the CNN+LSTM multimodal model, reinforcing the trends described above. Together, these results validate the dataset as a rich benchmark for developing and evaluating computer vision algorithms in physical rehabilitation, spanning objective kinematic analysis and clinically oriented quality assessment. Comparison of MAE (in degrees) for the four visual modalities (silhouettes, OF GMFlow, OF TV-L1, and semantic segmentation) across the four exercise pairs. The left plot shows results when training and testing on separated viewpoints, while the right plot reports the performance when training with all viewpoints. Usage Notes All code used in the experiments reported in this paper is available in the project repository24. It includes the definition of the splits used for training, validation, and test, and all the necessary configurations to replicate the models presented in this paper. The main functionalities of the provided code include: Generation of customized data manifests under different configurations, including custom splits and filtering by modality, camera, exercise, or subject. Generation of customized data manifests under different configurations, including custom splits and filtering by modality, camera, exercise, or subject. Training models for the 3D estimation of the angles of the joints that are performing the specified exercises. Training models for the 3D estimation of the angles of the joints that are performing the specified exercises. Evaluation of the trained models using MAE as a metric for comparison. Evaluation of the trained models using MAE as a metric for comparison. Easy configuration for hyper-parameter tuning via configuration files. Easy configuration for hyper-parameter tuning via configuration files. The provided code can also be modified to apply the same methodology to different datasets or to new model architectures. Limitations This study has several limitations that should be acknowledged. The available ground-truth MoCap data are restricted to a single joint per exercise, specifically the articulation directly involved in the prescribed movement. While this design is consistent with the clinical focus of the rehabilitation protocol, it prevents a comprehensive evaluation of multi-joint kinematics or whole-body coordination. As a result, the reported errors quantify performance only for the targeted degree of freedom, and do not allow us to assess how well the proposed modalities capture compensatory strategies or complex motor patterns involving additional joints. The current study has demographic limitations, including a limited sample size and gender imbalance. Furthermore, the dataset comprises healthy subjects, a deliberate choice to establish a normative ‘gold standard’ for correct exercise execution. While this provides a technical baseline for rehabilitation monitoring, we acknowledge that models trained on healthy movements may not immediately capture the compensatory patterns or reduced mobility observed in real patients. Future work will focus on including clinical populations to validate the transferability of these models to real-world postoperative scenarios. Furthermore, all derived modalities (e.g., silhouettes, optical flow, and semantic segmentation) are generated using pre-trained models developed on external datasets. Any systematic errors, biases, or domain shifts present in these models are propagated into our data and cannot be quantitatively isolated, since there is no independent ground-truth available for each representation. This should be taken into account when interpreting the results, and motivates future work aimed at incorporating modality-specific annotations or calibration procedures to better characterize and control these sources of error. Acknowledgements We are grateful to Manuel García and Clara de Araujo for annotating the exercise repetitions. This work has been partially supported by the Spanish Ministry of Science, Innovation, and Universities, under project PID2023-147296NB-I00. Author contributions Rafael Aguilar-Ortega was responsible for the data acquisition and analysis, methodology development, and conducting the experiments. Jorge Zafra-Palma contributed to multi-modal calculations and experimental setup. Rafael Muñoz-Salinas and Manuel J. Marín-Jimenez supervised the project, provided guidance, and assisted in manuscript writing and results analysis. Data availability The UCOPhyRehab++ dataset is publicly available on Zenodo15 with 10.5281/zenodo.17935736. The dataset is released under an open license. Code availability The code used for baseline experiments is publicly available on24. The repository includes documentation and scripts required to reproduce the results reported in this study. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The UCOPhyRehab++ dataset is publicly available on Zenodo15 with 10.5281/zenodo.17935736. The dataset is released under an open license. The code used for baseline experiments is publicly available on24. The repository includes documentation and scripts required to reproduce the results reported in this study.