An Adaptive IMU–Visual Multimodal Fusion System for Real-Time Exercise Recognition and Movement Quality Assessment Authors: Zhaoyang Gu, Ruopeng Yang, Yongqi Shi, Dongxu Dai, Chaoyang Li, Bo Huang, Kaige Jiao, Yu Tao, Yongqi Wen, Yihao Zhong, Chen He Published: 2026 ABSTRACT Exercise recognition and movement quality assessment remain challenging in supervised exercise training, particularly under viewpoint changes and self-occlusion. Vision-based methods provide spatial posture information but are susceptible to keypoint loss, whereas inertial sensing is less affected by occlusion but provides limited information about global posture geometry. This study presents a dual-stream prototype that combines a nine-axis inertial measurement unit (IMU) with vision-based pose estimation. A 1DCNN-LSTM branch models inertial dynamics, a custom keypoint temporal branch models normalized pose sequences, and a confidence-gated rule adjusts their contributions according to visual keypoint reliability. Owing to the absence of a public synchronized multi-view IMU–vision exercise dataset with the required protocol, we constructed IMV-Exercise, comprising 10 participants, three exercises, and 900 repetition-level samples with side-, front-, and posterior-view recordings. The system achieved 96.0% exercise recognition accuracy under leave-one-subject-out cross-validation. In a separate viewpoint-specific evaluation, the fused output achieved 91.2% action-window accuracy under posterior viewing. Across 50 online trials, the reported recognition accuracy was 96.0%, the mean end-to-end latency was 195 ms, and the recorded maximum was below 210 ms. These results establish feasibility within the studied cohort and exercises; broader generalization and feedback effectiveness require larger, independently controlled evaluations. 1. Introduction In structured or supervised exercise training settings, participants repeatedly perform standardized movements, and coaches or instructors are expected to monitor movement quality and provide timely corrective feedback. During high-frequency or prolonged training, direct observation alone may not provide consistent and quantitative assessments of every repetition, particularly with respect to movement range, trunk stability, joint compensation, and exercise rhythm. Wearable sensors have consequently been investigated for real-time kinematic monitoring and training support across sports and exercise applications [1,2,3,4,5,6,7,8,9]. Such settings allow a lightweight wearable IMU to be fitted before a training session and used together with a camera, making the joint deployment of inertial and visual sensing feasible. Accordingly, the intended application of this study is structured training assistance, in which wearable inertial sensing and vision-based pose estimation are jointly employed for exercise recognition and movement quality assessment. Existing methods for exercise recognition and movement quality assessment primarily rely on either vision-based or wearable inertial sensor approaches. Vision-based approaches typically use RGB cameras, depth sensors, or human pose estimation algorithms to capture keypoints and then apply spatiotemporal graph models, convolutional networks, Transformers, or other deep learning architectures for exercise recognition [10,11,12,13,14,15,16]. These methods provide rich spatial posture information, making them suitable for analyzing joint angles, limb trajectories, and overall posture deviations [13,14,15]. However, their performance is sensitive to camera viewpoint, lighting variations, background clutter, and limb self-occlusion [10,12,17]. In exercises such as push-ups, sit-ups, and squats, when participants adopt low-body positions, are viewed from the posterior, or experience trunk occlusion, keypoints such as the shoulders, elbows, wrists, hips, and knees may be partially or entirely undetected, reducing the reliability of exercise recognition and posture assessment. Compared with vision-based approaches, wearable inertial-sensor-based methods do not depend on the camera viewpoint, and offer advantages such as low power consumption, ease of deployment, and greater robustness to occlusion [9,18,19,20,21,22,23,24,25,26,27,28]. Inertial measurement units (IMUs) can collect time-series data, including acceleration, angular velocity, and attitude angles, which are suitable for characterizing movement rhythm, acceleration peaks, rotational states, and periodic dynamic patterns. Traditional inertial-sensor-based recognition methods usually rely on manually designed time-domain or frequency-domain features combined with classifiers such as support vector machines and hidden Markov models [29]. Recent inertial-sensing approaches include model-based joint-motion estimation and deep temporal models for learning movement representations from inertial sequences [5,30,31,32,33,34,35,36,37,38]. However, single-node or sparse inertial sensors mainly capture local dynamic changes and provide limited global spatial posture constraints. As a result, they remain limited in detecting fine-grained posture deviations, such as insufficient elbow flexion, trunk inclination, and knee valgus. The analysis above indicates that the visual and inertial modalities provide complementary information in real training scenarios. The visual modality offers spatial posture information but is less reliable under occlusion or varying viewpoints. In contrast, inertial sensors provide continuous dynamic information but offer limited insight into global posture relationships. Prior work on multimodal and multi-sensor fusion, including combinations of video and inertial sensing, illustrates the value of complementary measurements [39,40,41,42,43,44,45,46,47,48]. Nevertheless, real-time multimodal exercise systems still face three primary challenges. First, the sampling frequencies and data structures of different modalities differ, requiring temporal alignment. Second, multimodal models incur additional computational cost, requiring a balance between recognition performance and responsiveness. Third, a training assistance system must quantify movement completion, posture deviations, and rhythm stability in addition to classifying the exercise. To address these challenges, this study presents a dual-stream movement quality assessment prototype that combines a nine-axis IMU with vision-based pose estimation. The system synchronizes IMU signals and visual keypoint sequences through timestamp-based alignment and linear interpolation, and uses pre-extracted human keypoint sequences instead of raw RGB tensors as the temporal visual model input. Metrics based on Dynamic Time Warping (DTW) and Euler-angle stability quantify movement completion, posture deviation, and rhythm variation. A confidence-gated fusion rule then adjusts modality weights according to mean visual keypoint confidence and generates an exercise class together with rule-based corrective cues. The main contributions of this study are summarized as follows:A confidence-gated IMU–visual fusion pipeline is developed for exercise recognition and posture assessment. It combines temporal information from nine-axis IMU signals with spatial information from visual keypoint sequences and changes the visual contribution according to mean keypoint confidence.Movement quality assessment metrics based on Dynamic Time Warping (DTW) and Euler-angle stability are introduced to quantify movement completion, posture deviation, and rhythm variation.The IMU–Vision Multi-View Exercise Dataset (IMV-Exercise), a self-collected IMU–visual multimodal exercise dataset, is constructed to support the development and evaluation of the proposed system.A real-time training assistance prototype is implemented and evaluated for exercise recognition, viewpoint-dependent visual failure, and aggregate end-to-end latency within the three-exercise IMV-Exercise protocol. A confidence-gated IMU–visual fusion pipeline is developed for exercise recognition and posture assessment. It combines temporal information from nine-axis IMU signals with spatial information from visual keypoint sequences and changes the visual contribution according to mean keypoint confidence. A confidence-gated IMU–visual fusion pipeline is developed for exercise recognition and posture assessment. It combines temporal information from nine-axis IMU signals with spatial information from visual keypoint sequences and changes the visual contribution according to mean keypoint confidence. Movement quality assessment metrics based on Dynamic Time Warping (DTW) and Euler-angle stability are introduced to quantify movement completion, posture deviation, and rhythm variation. Movement quality assessment metrics based on Dynamic Time Warping (DTW) and Euler-angle stability are introduced to quantify movement completion, posture deviation, and rhythm variation. The IMU–Vision Multi-View Exercise Dataset (IMV-Exercise), a self-collected IMU–visual multimodal exercise dataset, is constructed to support the development and evaluation of the proposed system. The IMU–Vision Multi-View Exercise Dataset (IMV-Exercise), a self-collected IMU–visual multimodal exercise dataset, is constructed to support the development and evaluation of the proposed system. A real-time training assistance prototype is implemented and evaluated for exercise recognition, viewpoint-dependent visual failure, and aggregate end-to-end latency within the three-exercise IMV-Exercise protocol. A real-time training assistance prototype is implemented and evaluated for exercise recognition, viewpoint-dependent visual failure, and aggregate end-to-end latency within the three-exercise IMV-Exercise protocol. The remainder of this manuscript is organized as follows. Section 2 describes the system architecture, hardware platform, data acquisition protocols, inertial and visual data processing methods, dual-stream models, multimodal fusion strategy, and movement-quality outputs. Section 3 presents the experimental settings, recognition outcomes, baseline comparisons, viewpoint-dependent visual failure analysis, and aggregate online latency measurements. Section 4 interprets the results and addresses limitations, and Section 5 summarizes the key findings. Appendix A reports a separate inertial-only reproducibility check on the public MEx dataset. 2. Materials and Methods This section describes the system architecture, data acquisition protocol, signal preprocessing, model implementation, multimodal fusion strategy, feedback mechanism, and evaluation metrics. The description reports the implementation details that can be verified from the retained experimental specification. General theoretical derivations of established methods, such as MEMS sensing, 1DCNN, LSTM, and DTW, are omitted. 2.1. System Overview The overall system workflow is as follows. First, the JY901 nine-axis IMU (WitMotion, Shenzhen, China) collects inertial data, which are transmitted to a local workstation via an ESP8266-based WiFi module (ESP8266 chip: Espressif Systems, Shanghai, China). Simultaneously, an RGB camera captures the video stream, and human keypoints are extracted on the processing unit using MediaPipe Pose [49,50]. The system then preprocesses the IMU data and keypoint sequences and aligns them using timestamps recorded on the processing unit. The IMU branch employs a 1DCNN-LSTM model, while a custom temporal encoder operating on keypoint sequences forms the visual branch. A confidence-gated rule combines the two branch representations and supplies the exercise class to the movement-quality and feedback modules. The overall system architecture is illustrated in Figure 1. The IMU input vector at sampling time t is defined as(1)ximu(t)=[ax(t),ay(t),az(t),ωx(t),ωy(t),ωz(t),ϕ(t),θ(t),ψ(t)]T, where ax,ay,az denote the three-axis acceleration, ωx,ωy,ωz denote the three-axis angular velocity, and ϕ,θ,ψ denote the roll, pitch, and yaw angles, respectively. The visual input consists of human keypoints estimated by MediaPipe Pose. The visual input at frame t is defined as(2)Xvis(t)={k1(t),k2(t),…,kJ(t)},J=17, where kj(t) represents the normalized coordinates and detection confidence of the j-th core keypoint. The exercise class output is defined as(3)y^∈{push-up,sit-up,squat}, and the corresponding posterior probability vector is given by(4)p=[ppush,psit,psquat]. The movement quality outputs include movement completion, posture stability, and rhythm deviation. The corrective feedback labels include “good completion”, “insufficient depth”, “torso deviation”, “knee valgus”, and “tempo too fast”. 2.2. Hardware Platform and Data Acquisition The hardware platform, illustrated in Figure 2, consists of a JY901 nine-axis MEMS inertial sensor (WitMotion, Shenzhen, China), an ESP8266 WiFi communication module, a 3.7 V 2500 mAh lithium-polymer battery, a 1920 × 1080 RGB camera, and a local workstation. In the experimental configuration, the JY901 stream was acquired at 200 Hz and provided three-axis acceleration, three-axis angular velocity, and roll–pitch–yaw angles. The protocol conversion ranges were ±16g for acceleration and ±2000∘/s for angular velocity; roll and yaw were represented over ±180∘ and pitch over ±90∘. The RGB stream was recorded at 30 FPS. These values describe the configuration used in this study rather than a performance validation of the sensor itself. The ESP8266 module was used only for wireless transmission of inertial data and did not perform deep learning inference. The inertial data were transmitted to the processing unit via the TCP protocol, and CRC16 checksums were used to detect packet errors. Packets that failed checksum validation were excluded from the subsequent model input buffer. A data buffer queue was implemented to temporarily store inertial data during short-term network fluctuations, with a maximum capacity of 500 sampling records. Model inference, cross-modal fusion, and feedback decision-making were all performed on the local workstation. 2.3. Participants and Experimental Protocol Given the limited availability of public synchronized multi-view IMU–vision exercise datasets for supervised exercise training, this study constructed the IMU–Vision Multi-View Exercise Dataset (IMV-Exercise). A total of 10 participants were recruited for data collection, including 6 males and 4 females. Participant age, height, body weight, and the number of exercise samples are summarized in Table 1. IMV-Exercise included three exercise types: push-ups, sit-ups, and squats. Each participant performed 30 standardized repetitions for each exercise type, resulting in 90 exercise samples per participant. In total, IMV-Exercise contained 900 exercise samples. Model generalization performance was evaluated using leave-one-subject-out (LOSO) cross-validation. In each iteration, data from nine participants were used for training, and the remaining participant’s data served as an independent test set. This process was repeated for all ten participants. This validation scheme prevents data from the same participant from appearing in both the training and test sets, thereby reducing the risk of data leakage and providing an assessment of the model’s ability to generalize to unseen participants. To evaluate the recognition capability of the visual modality across different viewpoints in IMV-Exercise, exercise videos were collected from side, front, and posterior views. For each exercise–viewpoint pair, 1000 frames were sampled from the recorded viewpoint-specific videos after excluding transition frames before and after exercise execution. These frames were then processed frame by frame to count correctly recognized frames, misclassified frames, and frames for which no exercise label was produced. The cohort was small and demographically narrow (21.0±1.4 years), and only three standardized periodic exercises were included. Consequently, LOSO evaluation measures participant-independent performance within this sampled population; it does not establish generalization to other age groups, clinical populations, exercise types, or unconstrained environments. 2.4. IMU Data Preprocessing and Segmentation The raw IMU data consist of three-axis acceleration, three-axis angular velocity, and three-axis Euler angles. Due to differences in scale and range among channels, as well as potential disturbances from sensor movement and transient noise during acquisition, the system first performs smoothing, differential feature extraction, standardization, and action segmentation on the raw signals. To reduce the impact of high-frequency noise on action boundary detection, a moving average filter is first applied to the raw sequence: (5)y[n]=1N∑k=0N−1x[n−k], where x[n] and y[n] represent the discrete signals before and after filtering, respectively, and N is the filter window length. Subsequently, the system computes multi-axis differential energy to quantify short-term motion changes. The differential step is set to τ=3, and the differential energy is defined as(6)Δd[n]=∑m∈{x,y,z}sm[n]−sm[n−τ], where sm[n] denotes the signal along axis m at frame n. For acceleration, angular velocity, and angle signals, the system calculates the corresponding three-axis differential energy to enhance dynamic changes at the start and end of each movement. The segmented action windows are then standardized using Z-score normalization: (7)zi=xi−μσ, where xi is the i-th sample of a given channel, μ and σ are the mean and standard deviation of the channel within the current action segment, and zi is the normalized output. This processing reduces the influence of differing physical units and channel amplitude ranges on model training. For the continuous IMU data stream, movement segmentation was performed using a sliding window combined with a multi-threshold decision scheme. The sliding-window length was set to 40 frames with a step size of 20 frames. Based on sample statistics and sensor characteristics, the differential thresholds were set as follows: 0.8m/s2 for acceleration, 160∘/s for angular velocity, and 50∘ for angles. When any signal within the window exceeded its corresponding threshold, a candidate action segment was triggered. The segment start was determined by searching 20 frames to the left of the window center, and the end was determined by searching 20 frames to the right. After segmenting an action, the left boundary of the sliding window was updated to the current action endpoint to continue scanning the subsequent data stream. If no threshold was exceeded within the window, the entire window shifted 40 frames forward. The action segmentation workflow is illustrated in Figure 3. The rule-based detector was used to initiate inference during continuous online operation. By contrast, the offline LOSO evaluation used the 900 collected repetition-level samples after action-window construction; therefore, the reported 96.0% recognition accuracy is a classification metric and should not be interpreted as an end-to-end segmentation score. Independent frame-level onset and offset annotations were not retained for IMV-Exercise, so segmentation precision, recall, and boundary-error tolerance cannot be computed from the available records. Boundary shifts may include transition samples or truncate movement phases, and zero padding standardizes input length but does not remove this source of error. The absence of an independent segmentation evaluation is addressed as a limitation in Section 4. 2.5. Vision-Based Pose Estimation The visual modality supplements the spatial posture information that is not captured by the IMU signals. Human keypoints were extracted from RGB video frames using MediaPipe Pose, which initially outputs 33 keypoints. Considering that the exercises in this study primarily involve upper-limb support, trunk flexion, and lower-limb squats, 17 core keypoints—including the shoulders, elbows, wrists, hips, knees, and ankles, together with additional head and foot reference points—were selected from the original set for subsequent temporal modeling in the visual branch. For each image frame, the image coordinates and confidence score of the j-th keypoint are defined as(8)kj(t)=[uj(t),vj(t),cj(t)]T, where uj(t) and vj(t) denote the normalized two-dimensional image coordinates, and cj(t) denotes the keypoint detection confidence provided by MediaPipe Pose. To reduce the influence of participant body size and camera distance on the posture sequence, the keypoint coordinates were normalized using the hip center as the local coordinate origin: (9)p˜j(t)=pj(t)−phip(t), where pj(t)=[uj(t),vj(t)]T represents the two-dimensional coordinates of the j-th keypoint, and phip(t) represents the center coordinate of the left and right hip joints. The normalized keypoint sequence forms the input to the visual branch:(10)Xvis∈RTv×17×3, where Tv is the number of frames in the visual window. The three entries for each of the 17 selected keypoints are the hip-centered horizontal coordinate, the hip-centered vertical coordinate, and the MediaPipe confidence score. Because the IMU sampling frequency is 200Hz and the video frame rate is 30FPS, the two modalities have different temporal resolutions. To avoid temporal mismatch caused by direct concatenation, the system aligns the two data streams using timestamps recorded on the processing unit. Let the IMU timestamp set be(11)Timu={t1,t2,…,tM}, and the video frame timestamp set be(12)Tvis={τ1,τ2,…,τN}. For each IMU sampling time ti, the nearest video frame timestamp τζ(i) is selected as(13)ζ(i)=argminj|ti−τj|. When a visual keypoint sequence with the same temporal resolution as the IMU samples is required, linear interpolation is performed between adjacent video frames. If the normalized coordinates of a keypoint at adjacent video frames τk and τk+1 are p(τk) and p(τk+1), respectively, then the interpolated coordinate at t∈(τk,τk+1) is given by(14)p*(t)=p(τk)+t−τkτk+1−τkp(τk+1)−p(τk). This temporal alignment procedure was used to construct synchronized cross-modal action windows. Linear interpolation only estimates keypoint coordinates between two observed 30 FPS frames; it does not create additional visual evidence. It is a local constant-velocity approximation over an inter-frame interval of approximately 33.3 ms and can be inaccurate during rapid direction changes or when either neighboring keypoint is poorly localized. The interpolated confidence was therefore retained as a reliability cue for fusion, but no claim of sub-frame pose accuracy is made. Subsequent model training and online testing were performed based on the aligned action windows. 2.6. 1DCNN-LSTM Branch The IMU branch employs a 1DCNN-LSTM model to extract temporal dynamic features from the motion data. For each successfully segmented action, the nine-axis IMU data are organized as a multi-channel temporal sequence: (15)Ximu∈RT×C,C=9, where T is the length of the action window and C is the number of IMU channels. The nine channels correspond to three-axis acceleration, three-axis angular velocity, and three-axis Euler angles. Action segments of different lengths are zero-padded along the temporal dimension to obtain a uniform sequence length Lmax. The padded IMU input is represented as(16)X˜imu∈RLmax×C,C=9. The IMU branch follows the sequence shown in Figure 4: a first temporal convolution block, global max pooling, a second temporal convolution block with dropout, an LSTM layer, and a three-class Softmax layer. The convolution blocks extract local waveform changes, whereas the LSTM models dependencies among movement phases. The representation supplied to the fusion module is 128-dimensional. The retained experimental specification does not preserve the convolutional kernel widths, filter counts, stride and padding settings, or LSTM hidden dimension. These values are therefore not inferred from a different preliminary implementation, and this reproducibility limitation is stated explicitly in Section 4. The IMU branch outputs posterior probabilities for the three exercise classes:(17)pimu=Gimu(X˜imu), where Gimu(·) denotes the 1DCNN-LSTM IMU branch. The input reconstruction and 1DCNN-LSTM branch structure are illustrated in Figure 4. The model was trained using a multi-class cross-entropy loss function: (18)Lce=−∑k=1Kyklog(p^k), where K=3 is the number of classes, yk is the one-hot encoded ground-truth label for class k, and p^k is the predicted posterior probability for class k. 2.7. Keypoint Temporal Branch The visual branch is a custom action-window classifier operating on normalized keypoint sequences. Its use of multiscale temporal representation and local temporal attention was conceptually informed by ActionFormer [51], but the branch is neither the original ActionFormer temporal-localization model nor an implementation of its RGB/video-feature front end. The input has the shape given in Equation (10) and comprises keypoint sequences after hip-center normalization and temporal alignment. At each frame, the 17 triplets form a 51-dimensional keypoint vector. MediaPipe performs keypoint extraction before this branch, so neither raw RGB tensors nor the original ActionFormer video-feature front end enter the classifier. The retained implementation interface records a 256-dimensional visual representation before fusion and one exercise-class output for each presegmented action window. It does not preserve the fine-grained projection, temporal-convolution, attention-head, or pooling settings. The branch is therefore reported at the verifiable input–output and block level, without claiming exact source-level reproduction of its internal encoder. The visual branch outputs the posterior probabilities of the exercise classes:(19)pvis=Gvis(Xvis), where Gvis(·) denotes the adapted keypoint temporal branch. The visual branch used Focal Loss [52], which down-weights easily classified samples during optimization:(20)Lfocal=−αt(1−p^t)γlog(p^t), where p^t is the predicted probability corresponding to the ground-truth class, αt is the class weighting factor, and γ is the focusing parameter used to down-weight easily classified samples. 2.8. Confidence-Gated Multimodal Fusion The IMU and visual modalities provide complementary information. The IMU modality does not rely on camera viewpoints and can deliver dynamic information under posterior-view and self-occlusion conditions. The visual modality provides spatial relationships between joints but becomes less reliable when keypoints are occluded or their confidence decreases. The implemented fusion is a low-cost confidence-gated heuristic rather than a learned uncertainty model. Let fimu denote the high-level features output by the 1DCNN-LSTM IMU branch, and fvis denote the high-level features from the keypoint temporal branch. Before fusion, both modalities are projected to a unified dimensionality via linear mappings:(21)f˜imu=fimuWimu,(22)f˜vis=fvisWvis, where Wimu and Wvis are learnable projection matrices. The fused feature is defined as(23)Ffusion=wimuf˜imu⊕wvisf˜vis, where ⊕ denotes feature concatenation, and wimu and wvis are the fusion weights for the IMU and visual branches, respectively. The visual confidence cvis is computed as the average detection confidence of the 17 core keypoints within the current action window: (24)cvis=1TvJ∑t=1Tv∑j=1Jcj(t), where Tv is the number of visual frames in the window, J=17 is the number of core keypoints, and cj(t) is the confidence of the j-th keypoint in frame t. The fusion weights are determined by(25)wvis=cvis,wimu=1−cvis,cvis≥γth,wvis=0,wimu=1.0,cvis<γth, where γth=0.45 is the fixed visual-confidence threshold used in the implementation. When cvis falls below this threshold, the visual contribution is disabled; otherwise, the two weights vary linearly with the mean confidence. All 17 joints contribute equally to cvis. Thus, the rule has a discontinuity at γth and does not encode exercise-specific joint importance. The retained evaluation did not include a threshold-sensitivity study, so 0.45 is treated only as the fixed engineering setting used by this prototype, not as an optimized or generally applicable value. The final fused classification result is given by(26)y^=Gfusion(Ffusion), where Gfusion(·) represents the fusion classification module. The three-layer cascaded multimodal fusion topology is illustrated in Figure 5. The first layer performs cross-modal temporal alignment. The second layer executes feature-level adaptive fusion based on visual keypoint confidence. The third layer generates corrective feedback labels by combining movement quality metrics with predefined feedback rules. The third layer in Figure 5 is a rule-based feedback-priority module, not a trained “biomechanical logic tree”. It receives the predicted exercise class together with posture and temporal quality flags. If a configured safety flag is present (the prototype includes a squat knee-valgus flag at θknee≥15∘), the corresponding safety cue has first priority. If no safety flag is present, completion/posture cues derived from the DTW and stability metrics have second priority, followed by rhythm cues. If no threshold is exceeded, no corrective cue is issued. At most one cue is emitted for each action window. These rules generate user-facing labels and are not included in the reported exercise-recognition accuracy. 2.9. Real-Time Feedback Strategy System feedback is determined jointly by the exercise classification results and the movement quality assessment. Once an action segment is segmented, the system first outputs the exercise class along with its posterior probability. Subsequently, corrective feedback labels are generated based on movement completion, posture stability, and rhythm deviation. Feedback labels are categorized into three priority classes: safety, correctness, and rhythm. Safety-category feedback denotes a predefined posture-rule violation, such as apparent medial knee deviation, pronounced trunk deviation, or unstable wrist support; the label is not a clinical injury-risk assessment. Correctness feedback highlights insufficient movement amplitude, for example, inadequate squat depth, incomplete push-up lowering, or partial sit-up completion. Rhythm feedback addresses timing or speed issues, such as movements performed too quickly, too slowly, or with abnormal pauses between phases. When multiple flags occur simultaneously, the system outputs the highest-priority label according to the order: safety, correctness, and rhythm. Each successfully segmented action window triggers at most one primary feedback label. Posture stability is characterized by the variation in Euler angles within an action cycle window. Let the action cycle window be W, and let the instantaneous posture angle vector be Θ(t)=[ϕ(t),θ(t),ψ(t)]T. The posture stability metric is defined as(27)σΘ=1|W|∑t∈WΘ(t)−Θ¯22, where Θ¯ denotes the mean posture angle vector within the window. A larger σΘ indicates greater trunk or limb posture fluctuation during movement execution. Movement completion and rhythm deviation are evaluated using Dynamic Time Warping (DTW) [2,53]. Let the feature sequence of the action to be evaluated be Y=(y1,y2,…,yM), and let the standard action template sequence be Q=(q1,q2,…,qN). The normalized DTW distance between the two sequences is defined as(28)D¯DTW(Y,Q)=1KminW∑k=1Kyik−qjk22, where W={(ik,jk)}k=1K is the optimal temporal alignment path between the test sequence and the template sequence, and K is the path length. The value of D¯DTW is used to quantify the overall difference between the performed movement trajectory and the standard template. If this distance exceeds a predefined threshold, the system identifies the movement as incomplete or deviating from the standard trajectory. Local temporal stretching or compression along the alignment path is further used to detect movements that are too fast, too slow, or contain abnormal pauses between phases. The feedback results are displayed through a PyQt5-based visualization interface and can also be converted into voice prompts using a text-to-speech (TTS) module. During online operation, the feedback trigger time is recorded for subsequent end-to-end latency analysis. 2.10. Evaluation Metrics and Measurement Scope Exercise recognition performance was evaluated using accuracy, precision, recall, and F1 score. For this three-class classification task, precision, recall, and F1 score were computed for each class and then averaged to obtain the macro metrics. Let TPk, FPk, and FNk denote the true positives, false positives, and false negatives, respectively, for class k, with a total of K=3 classes. The macro-averaged precision, recall, and F1 score are defined as(29)Precisionmacro=1K∑k=1KTPkTPk+FPk,(30)Recallmacro=1K∑k=1KTPkTPk+FNk,(31)F1macro=1K∑k=1K2·Precisionk·RecallkPrecisionk+Recallk. Overall accuracy is defined as(32)Accuracy=NcorrectNtotal, where Ncorrect is the number of correctly classified samples, and Ntotal is the total number of test samples. For the visual-modality viewpoint experiment, the numbers of correctly recognized frames, misclassified frames, and unrecognized frames were calculated separately for the side, front, and posterior views using the 1000-frame subsets sampled for each exercise–viewpoint pair. Frame-level accuracy was computed as the proportion of correctly recognized frames among all test frames, with unrecognized frames treated as incorrect predictions. Precision among recognized frames was computed as the proportion of correctly recognized frames among frames that received a valid class prediction. The unrecognized-frame rate was computed as the proportion of frames for which no valid class prediction was produced. To evaluate whether the system meets the requirements for online training feedback, end-to-end latency was defined as the time interval from the arrival of the last required input frame for the current action window at the processing unit to the generation and triggering of the feedback output. For the i-th online inference event, the end-to-end latency is defined as(33)Te2e(i)=tfb(i)−tin(i), where tin(i) denotes the timestamp at which the last IMU sample and its corresponding video frame for the i-th action window arrive at the processing unit, and tfb(i) denotes the timestamp at which the feedback label is written to the interface or the voice prompt is triggered. The end-to-end latency is further decomposed as(34)Te2e=Tpre+Talign+Tinfer+Tdecision+Tfeedback, where Tpre is the time required for data parsing, filtering, and action segmentation; Talign is the time required for cross-modal temporal alignment; Tinfer is the inference time of the 1DCNN-LSTM and keypoint temporal models; Tdecision is the time required for fusion decision-making and feedback-label generation; and Tfeedback is the time required for interface updating or voice-prompt triggering. The deployment logger stored only tin(i) and tfb(i); it did not retain the five internal timestamps. Consequently, the experiment supports aggregate end-to-end latency only, not a stage-wise latency comparison. A total of Nlat=50 online inference events were recorded in the latency test. For each event, tin(i) and tfb(i) were recorded, and the average end-to-end latency was calculated as(35)T¯e2e=1Nlat∑i=1NlatTe2e(i), The peak latency was reported as the maximum observed value among the 50 online inference events. 3. Results This section reports the experimental settings and the results obtained on IMV-Exercise. The primary evidence comprises leave-one-subject-out cross-validation, recognition-only baseline comparisons, a frame-level analysis of visual failures across viewpoints, viewpoint-specific fused-output accuracy, and aggregate online deployment measurements. An auxiliary inertial-only reproducibility check on the public MEx dataset is reported separately in Appendix A; it is not treated as an evaluation of the full IMU–visual system. 3.1. Experimental Settings Implementation Details. All offline model training and evaluation were conducted using a Python-based deep learning implementation on the local workstation used for system development. The preprocessing, temporal alignment, and action-window construction procedures described in Section 2 were applied to the corresponding inputs used by the IMU branch, visual branch, and fusion model. In each leave-one-subject-out (LOSO) fold, data from one participant were held out exclusively for testing, while data from the remaining nine participants were used for model fitting and validation. No samples from the held-out participant were used for parameter fitting or model selection. Implementation Details. All offline model training and evaluation were conducted using a Python-based deep learning implementation on the local workstation used for system development. The preprocessing, temporal alignment, and action-window construction procedures described in Section 2 were applied to the corresponding inputs used by the IMU branch, visual branch, and fusion model. In each leave-one-subject-out (LOSO) fold, data from one participant were held out exclusively for testing, while data from the remaining nine participants were used for model fitting and validation. No samples from the held-out participant were used for parameter fitting or model selection. The input and model settings documented in the submitted experimental record are summarized in Table 2. The IMU branch used multi-class cross-entropy, whereas the visual branch used Focal Loss. The baseline HMM, SVM, and ST-GCN values were obtained under the participant-level LOSO protocol using their corresponding input modalities. Because complete baseline configurations and fold-level outputs were not retained, their results are treated as descriptive within-protocol references rather than independently reproduced or controlled component ablations. Baseline Algorithms. Three recognition baselines were included: HMM and SVM using inertial features, and ST-GCN using visual skeleton sequences. The submitted experimental record reports each baseline under the participant-level LOSO protocol. Fold-level baseline predictions and complete baseline configurations needed for independent reproduction were not retained. The comparison therefore concerns descriptive exercise-classification accuracy only; it does not compare movement-quality outputs, model capacity, or online latency and is not used as a controlled ablation. Baseline Algorithms. Three recognition baselines were included: HMM and SVM using inertial features, and ST-GCN using visual skeleton sequences. The submitted experimental record reports each baseline under the participant-level LOSO protocol. Fold-level baseline predictions and complete baseline configurations needed for independent reproduction were not retained. The comparison therefore concerns descriptive exercise-classification accuracy only; it does not compare movement-quality outputs, model capacity, or online latency and is not used as a controlled ablation. Datasets. The primary evaluation used IMV-Exercise, which contains 10 participants, 900 repetition-level samples, three exercise categories (push-ups, sit-ups, and squats), and side-, front-, and posterior-view recordings. MEx [54] contains 30 subjects performing seven physiotherapy exercises with two body-worn accelerometers, a depth camera, and a pressure mat. As MEx does not reproduce the sensing configuration or outputs of the proposed system, its accelerometer streams were used only for the separate reproducibility check in Appendix A. Datasets. The primary evaluation used IMV-Exercise, which contains 10 participants, 900 repetition-level samples, three exercise categories (push-ups, sit-ups, and squats), and side-, front-, and posterior-view recordings. MEx [54] contains 30 subjects performing seven physiotherapy exercises with two body-worn accelerometers, a depth camera, and a pressure mat. As MEx does not reproduce the sensing configuration or outputs of the proposed system, its accelerometer streams were used only for the separate reproducibility check in Appendix A. 3.2. Main Results The fusion system was evaluated on IMV-Exercise using LOSO cross-validation and achieved an overall repetition-level exercise-recognition accuracy of 96.0%. This value measures generalization to held-out participants from the same sampled population and three exercise classes; it should not be extrapolated to other populations or exercises. Table 3 reports recognition accuracy for the fusion system and the three within-protocol reference baselines. The numerical differences in Table 3 are descriptive. Because the baselines use different sensing inputs and representations, the table does not isolate the contribution of the confidence-gating rule or establish superiority under matched model capacity. 3.3. Per-Action and Per-View Performance To evaluate the recognition performance of the visual modality across different viewpoints, frame-by-frame recognition results were computed for three exercise types under side, front, and posterior views. As described in the experimental protocol, each exercise–viewpoint pair contained 1000 sampled test frames from the corresponding recorded video subset. The retained frame-count summary does not identify each participant’s contribution or the original sampling seed. Moreover, adjacent video frames are correlated observations. The frame counts are therefore used as a descriptive failure analysis and not as an independent-sample statistical comparison. The results are summarized in Table 4. Based on the frame-by-frame statistics in Table 4, frame-level recognition accuracy was computed by treating unrecognized frames as incorrect predictions. Under the side view, the frame-level accuracies for push-ups, sit-ups, and squats were 100.0%, 95.0%, and 99.2%, respectively. Under the front view, the corresponding accuracies were 93.2%, 88.5%, and 86.6%. Under the posterior view, the accuracies were 0.0%, 25.7%, and 66.2%, respectively. The corresponding micro-averaged accuracies were 98.1%, 89.4%, and 30.6% (Table 5). The error pattern was dominated by rejection rather than assignment to another exercise class. Of the 3000 posterior-view frames, 2013 were unrecognized, including all 1000 push-up frames, 675 sit-up frames, and 338 squat frames; only 68 frames were recorded as cross-class errors, all from the sit-up subset. This indicates that posterior self-occlusion mainly caused the pose pipeline to withhold a valid class rather than consistently confuse one exercise pair. The retained aggregate counts do not include the destination label for those 68 frames, so a class-resolved 3×3 confusion matrix cannot be computed from the available records. Figure 6 visualizes accuracy and precision among recognized frames. Rejection rates are reported in Table 5; the aggregate counts are not presented as a confusion matrix. 3.4. Viewpoint-Specific Fused Output The retained fused-decision summary reported accuracies of 100.0%, 95.0%, and 91.2% for the side-, front-, and posterior-view action-window subsets, respectively (Table 6). These action-window values are reported separately from the visual frame-level statistics above. The earlier manuscript placed frame-level visual results and action-window branch results in one table; because those quantities have different evaluation units and the retained IMU-only outputs could not be matched sample by sample to the viewpoint subsets, that comparison was removed. The present results consequently describe the behavior of the fused output across viewpoints but do not constitute a controlled modality ablation. The lower posterior-view value is consistent with the high visual rejection rate in Table 5. Likely contributing mechanisms include residual errors from partially visible joints, equal weighting of all 17 keypoints in the visual-confidence average, the abrupt gate at γth=0.45, and alignment error during rapid movement phases. These mechanisms are plausible failure sources rather than class-resolved causal estimates because per-sample confidence traces and matched single-branch predictions were not retained. 3.5. Latency and Deployment Performance To evaluate online operation, 50 exercise-detection trials were conducted with the complete pipeline running. The reported online recognition accuracy was 96.0%. The mean end-to-end response latency was 195 ms and the recorded maximum was below 210 ms (Table 7). Only aggregate input and feedback timestamps were retained, so dispersion statistics and stage-wise latency cannot be computed from the available log. The baseline algorithms were not integrated into the same acquisition and feedback pipeline; a baseline-latency comparison is therefore not reported. Figure 7a shows an example of online system operation for squat recognition, and Figure 7b shows an example for push-up recognition. In each example, the console output on the left displays the exercise class and feedback label generated by the system, while the pose estimation result on the right shows the detected visual keypoints. 4. Discussion This study developed a dual-stream prototype that integrates a nine-axis IMU with vision-based pose estimation for three periodic exercises. The fusion system achieved 96.0% repetition-level recognition accuracy under LOSO evaluation on IMV-Exercise. In the separate online test, the reported recognition accuracy was 96.0%, with a mean end-to-end latency of 195 ms and a recorded maximum below 210 ms. These results establish feasibility within the sampled young-adult cohort and protocol. They do not by themselves establish broad population generalization, superiority over every matched single-modality model, clinical benefit, or the effectiveness of the generated corrective cues. 4.1. Interpretation of the Fusion and Failure Results The fusion design is motivated by complementary sensing characteristics. IMU signals represent acceleration, angular velocity, and attitude changes without depending on camera visibility, whereas keypoint sequences provide image-plane joint geometry. The present experiment shows that the fused output remained above 90% across the three recorded viewpoints, but it does not isolate how much of that performance is attributable to each branch or to the confidence gate. A controlled modality ablation would require the IMU-only, visual-only, and fused models to be evaluated on the same action windows with the same target labels and metric. Those matched per-sample outputs were not retained, so the earlier cross-unit comparison was removed. The visual frame analysis provides a clearer failure signal. The unrecognized-frame rate increased from 1.9% in the side view to 67.1% in the posterior view, with all 1000 posterior push-up frames rejected. Only 68 posterior frames were assigned to an incorrect class; the remaining 2013 errors were rejections. Thus, loss of detectable or sufficiently confident keypoints, rather than systematic confusion between two exercise classes, was the dominant recorded visual failure. The fused action-window accuracy also decreased from 100.0% in the side view to 91.2% in the posterior view. Because the frame-level and action-window results have different units, their numerical difference is not an estimate of fusion gain. The posterior-view decrease also identifies conditions under which the gate can fail. Averaging confidence over all 17 joints can mask the loss of exercise-critical joints, and a window just above the fixed threshold still contributes visual features despite potentially unreliable local geometry. Temporal interpolation and action-boundary offsets can add further mismatch during rapid phases. These explanations are consistent with the implemented pipeline, but they remain hypotheses because joint-level confidence traces and class-resolved fused errors were not preserved. 4.2. Robustness Under Occlusion and Viewpoint Changes Occlusion robustness is a key consideration in the design of the proposed system. Experimental results indicate that the visual modality exhibits a pronounced performance decline under posterior-view conditions, particularly for push-ups and sit-ups, where core keypoints are frequently occluded by the trunk or the ground. This limitation is consistent with the fundamental constraints of visual pose estimation: when key joints such as the shoulders, elbows, wrists, hips, or knees are not visible or have reduced confidence, the visual temporal model cannot reliably capture the complete spatial posture trajectory. The IMU branch is unaffected by camera visibility and continues to supply motion signals under posterior-view or self-occlusion conditions. The confidence gate was designed to reduce the visual contribution when mean keypoint confidence falls below 0.45. The current data show that the complete pipeline remained operational under posterior viewing, but they do not quantify the counterfactual performance of a continuously weighted gate, an IMU-only fallback, or a learned uncertainty model on the same samples. The current fusion rule is therefore an engineering heuristic. It gives equal importance to all selected joints, although shoulders, elbows, and wrists are more informative for push-ups and hips, knees, and ankles are more informative for squats. It also changes discontinuously at 0.45, and no threshold-sensitivity analysis was available. Future work should compare this operating point with a prespecified threshold sweep, use exercise-dependent or learned joint weights, calibrate confidence, and replace the hard gate with a smooth transition or uncertainty-aware fusion rule. 4.3. Comparison with Baseline Methods The fusion system had the highest descriptive accuracy in the within-protocol comparison (96.0%), followed by HMM (94.8%), SVM (89.0%), and ST-GCN (76.2%). HMM and SVM used inertial representations, whereas ST-GCN used visual skeleton input. The results therefore compare complete recognition pipelines rather than isolating a single architectural component. The practical distinction of the prototype is its integration of inertial dynamics, visual posture representation, movement-quality metrics, and a feedback interface in one pipeline. However, the baseline table does not control for input information, parameter count, feature engineering, or computational budget. It cannot establish that the fusion rule itself is responsible for the observed ranking. Accordingly, the comparison should be interpreted only as a reference within IMV-Exercise and the reported LOSO split. A future controlled study should evaluate matched-capacity single-branch and fusion models on identical action windows, report per-fold uncertainty, and retain per-sample predictions for paired statistical analysis. 4.4. Deployability and Real-Time Feedback The 50 online trials yielded a mean end-to-end latency of 195 ms and a recorded maximum below 210 ms. Under the current hardware and software configuration, the prototype therefore generated its output within the reported sub-second interval after receiving the final input required for an action window. This measurement begins after the action window is available and should not be confused with the duration of the exercise repetition or the unvalidated action-boundary detection delay. The implementation has three practical characteristics. First, the ESP8266 module transmits IMU data while deep-model inference is performed on the local workstation. Second, the visual branch receives selected keypoint sequences rather than raw RGB tensors. Third, movement-quality metrics are mapped to a limited set of deterministic output labels. These design choices describe the implemented prototype; their individual effects on computation time were not isolated experimentally. The latency evidence remains limited. The logger retained only the input-arrival and feedback-trigger timestamps, so standard deviation, percentile statistics, and stage-wise times for preprocessing, alignment, keypoint extraction, branch inference, fusion, and interface output cannot be computed from the available records. The 50 trials also came from one implementation environment. Future evaluation should retain every internal timestamp and report the distribution of each stage across devices and operating conditions. 4.5. Feedback Scope and Practical Implications The implemented module maps classification and movement-quality outputs to one prioritized text or speech cue. The present evaluation demonstrates cue generation and records the time required to trigger the output; it does not evaluate whether participants subsequently improve their technique. The earlier exploratory group comparison lacked retrievable group sizes, randomization details, a prespecified assessment scale, rater agreement, confidence intervals, and statistical tests, and has therefore been removed from the Results section. A feedback-effectiveness claim would require a prospectively designed controlled study with transparent allocation, blinded assessment, inter-rater reliability, effect sizes, uncertainty intervals, and a defined primary outcome. From a practical perspective, the system is better suited as an assistive tool for coaches or trainees rather than a replacement for professional instruction. It provides continuous, quantifiable, and recordable information on exercise recognition and feedback, reducing the burden of manual observation in group training scenarios. However, for complex injury risk assessment, individualized training load adjustments, and medical evaluation, professional judgment that considers the participant’s physical condition remains essential. Beyond on-site group training, the sensor-and-keypoint pipeline may also support remote or hybrid practice-oriented instruction, in which trainees use accessible sensing devices to transmit movement descriptors and receive automated cues while an instructor reviews summarized performance. Related work has explored smartphones and exergame controllers as bring-your-own-device sources of kinematic data for online sport and exercise science activities, as well as vision-based exercise biofeedback for telerehabilitation [8,55]. This application remains prospective: deployment outside supervised laboratory conditions would require validation across heterogeneous consumer devices, network conditions, home environments, and instructor workflows. 4.6. Limitations and Future Work This study has several limitations. First, the retained final-model specification records the branch order, losses, input and fusion feature dimensions, optimizer, learning rate, batch size, training limit, early-stopping patience, and dropout rate, but not every convolutional kernel, filter count, stride, padding rule, recurrent hidden dimension, or keypoint-encoder attention setting. The method can therefore be assessed at the block and interface level, but exact source-level reproduction of the trained architecture is not possible from the retained record. Future releases should archive executable configurations, source code, model summaries, and fixed checkpoints together with the data-processing scripts. Second, IMV-Exercise contains only 10 participants with a narrow age distribution and 900 repetitions. Although LOSO separation prevents the held-out participant from entering model fitting, it does not cover broader variation in age, training status, body type, clinical condition, or exercise technique. Future work should include a larger and more diverse cohort and report uncertainty across participants rather than only pooled accuracy. Third, the current experiments include only three exercise types: push-ups, sit-ups, and squats. These exercises are highly periodic, making them suitable for IMU temporal modeling and visual pose modeling, but they do not represent the full range of training exercises. Movements such as lunges, pull-ups, burpees, weighted exercises, agility drills, or combination exercises exhibit substantial variations in movement speed, joint involvement, and occlusion patterns. Future research should expand the set of exercise types and evaluate the model’s generalization across different exercises, environments, and devices. Fourth, the current experiments used relatively fixed camera and sensor placements. Lighting, background complexity, multi-person occlusion, camera distance, sensor loosening, orientation changes, and wireless loss were not systematically varied. The viewpoint analysis isolates only three recorded camera directions and should not be treated as evidence for unconstrained deployment. Continuous acquisition of video-derived pose and wearable-sensor signals also raises privacy and data-protection considerations. Although the current pipeline uses selected keypoints rather than raw RGB tensors for model input, source video and synchronized motion traces can still contain identifying or sensitive behavioral information. Future deployments should therefore implement data minimization, consent specific to data reuse, encryption in transit and at rest, access controls, retention limits, and, where feasible, on-device processing or privacy-preserving motion representations. Visual animation of kinematic data provides one related approach for reducing exposure while retaining analytic utility [56]. Fifth, independent onset and offset annotations were not retained, so segmentation precision, recall, boundary error, and classification robustness to shifted action windows could not be measured. Zero padding makes the network input length uniform but does not correct missing movement phases or included transition samples. Future data collection should preserve frame-level boundaries and evaluate both segmentation metrics and classification under controlled temporal offsets. Sixth, linear interpolation between 30-FPS visual observations assumes locally constant keypoint velocity. It can be inaccurate during rapid reversals and cannot recover occluded visual evidence. Higher-rate cameras, motion-aware interpolation, or alignment models that propagate uncertainty should be compared in future work. Seventh, the current fusion rule uses equal joint contributions, a fixed discontinuous threshold, and no reported sensitivity analysis. The retained summaries also do not support a fair matched-window modality ablation or a class-resolved confusion matrix. These analyses require per-sample branch predictions and confidence traces and should be part of a preregistered evaluation protocol. Eighth, this study evaluates recognition, movement-quality computation, and output latency, but not long-term training benefit or injury incidence. Such outcomes require longitudinal intervention designs and validated exercise-science or clinical endpoints. In summary, the results support the feasibility of the integrated prototype within IMV-Exercise and the recorded online setting. Stronger claims about fusion superiority, segmentation robustness, feedback effectiveness, population generalization, or injury prevention require the controlled and data-complete evaluations outlined above. 5. Conclusions This study presents a dual-stream prototype that combines nine-axis IMU sensing with vision-based keypoint sequences for exercise recognition and movement-quality computation. On IMV-Exercise, the fusion system achieved 96.0% repetition-level accuracy under LOSO evaluation. In a separate 50-trial online test, the reported recognition accuracy was 96.0%, with a mean end-to-end latency of 195 ms and a recorded maximum below 210 ms. The viewpoint analysis also quantified a substantial increase in visual rejection under posterior viewing, while the fused action-window output remained at 91.2% accuracy. The evidence is limited to 10 young adults, three periodic exercises, fixed acquisition conditions, aggregate latency logs, and retained summary outputs. It does not establish a controlled modality advantage or feedback effectiveness. Future work should retain executable model specifications, per-sample predictions, and internal timestamps; validate segmentation and threshold sensitivity; use matched-window ablations; and evaluate larger and more diverse cohorts under realistic conditions.