Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

Francis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum · 2026

Read Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training by Francis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28104.txt.

Introduction

Traditional Chinese Medicine rehabilitation involves techniques like acupuncture and Tuina massage. Judging trainee performance visually is hard, because hands occlude each other and interact with objects. Francis Xiatian Zhang and six colleagues propose CME-AQA in 2026, a cross-view multimodal assessment framework. Visual and pose information are fused for context. Training uses both first-person and third-person videos, which improves robustness at inference. Two new datasets support the work.

Each dataset records sixty-one subjects in synced first- and third-person views, with expert scores. One covers acupuncture, the other Tuina. On key rating tasks like needle depth and quick needle insertion, the framework improves weighted F1 by over ten percent relative to the best competing method. Mean absolute error also falls for insertion time and manipulation frequency. Tests on a CPR dataset give comparable performance on several posture criteria.

Catalog record

CREATORFrancis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training Francis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum Abstract Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action quality assessment (AQA) from computer vision offers a promising solution. Existing automatic AQA frameworks for physical therapy typically rely on skeletal data captured from a single viewpoint, which is inefficient for TCM techniques such as acupuncture or Tuina that involve dense hand self-occlusion and complex hand–object interactions. To address these challenges, we propose CME-AQA, a cross-view, multimodal vision-based assessment framework that integrates visual–pose fusion to enhance understanding of environmental context and leverages both first-person and third-person videos during training to improve inference robustness. We collected two dual-view datasets, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina), each containing synchronized first- and third-person recordings of 61 subjects with expert annotations. Experimental results show that our approach achieves superior or comparable mean performance against competitive baselines , achieving over 10% relative improvement in weighted F1 over the best competing method on key rating tasks such as Needle Depth and Quick Needle Insertion, while also reducing mean absolute error in quantitative measures such as insertion time and manipulation frequency. Testing on a CPR dataset further demonstrates comparable performance on several posture-based criteria, suggesting applicability to related structured simulated clinical skill assessments where participant motion is central to evaluation. Overall, CME-AQA enhances assessment accuracy for structured TCM rehabilitation training and facilitates more convenient and effective training-oriented skill evaluation . Fig. 1: Example frames from our two collected datasets: acupuncture (upper rows) and Tuina (lower rows). Each session was recorded with two cameras: a forehead-mounted first-person view (left column) and a separate third-person view of the hands (right column), providing complementary participant and observer perspectives. I Introduction Traditional Chinese Medicine (TCM) rehabilitation therapies, such as acupuncture and Tuina (Chinese therapeutic massage), have demonstrated measurable effectiveness in treating motor and neurological disorders [ 1 , 2 ] . Recent advancements in TCM rehabilitation training [ 3 , 4 , 5 ] have introduced automatic assessment tools to evaluate performance in structured TCM rehabilitation training settings . These tools enhance accessibility to training, especially in developing regions where experienced instructors are scarce. However, many current systems rely on specialized equipment, such as VR glasses [ 5 ] or robotic platforms [ 4 ] , to assess practice behavior, which limits their practicality. To address this limitation, we propose a vision-based assessment framework that derives performance metrics directly from video, providing a cost-effective and portable solution. In human-performance assessment [ 6 ] , action quality assessment (AQA) is a widely adopted method for evaluating practice actions across multiple predefined aspects [ 7 ] . It applies computer vision techniques to quantify the quality of movements performed during training and has proven effective in both sports and rehabilitation contexts [ 8 , 6 ] . Most existing frameworks rely on pose-based skeletal data obtained through sensor-based or markerless motion capture to represent body configurations and dynamics, which helps reduce environmental noise and capture motion details accurately [ 9 ] . However, current AQA methods [ 10 , 11 , 12 ] mainly analyze full-body movements from a single-view perspective, which presents a major limitation for TCM rehabilitation tasks. These procedures require precise hand movements that are frequently obscured by self-occlusion [ 13 ] . Moreover, using only skeletal data omits key environmental cues that are critical for modeling hand–object interactions, for example, the manipulation of acupuncture needles on practice pads [ 14 ] . To design an effective assessment framework for TCM rehabilitation training, we identify three key challenges in existing AQA methods. First, current approaches [ 10 , 11 , 15 ] primarily depend on pose estimation data, which often fail to capture the environmental context where the hands interact with objects and surfaces during TCM procedures. This limitation reduces the capacity of AQA models to evaluate all clinically relevant aspects of rehabilitation performance. Second, existing AQA frameworks [ 16 , 17 ] , typically designed for single-view video inputs, struggle to handle frequent self-occlusions in which one hand or tool obscures another. Such occlusions can compromise the reliability and continuity of performance evaluation. Third, to the best of our knowledge, no publicly available dataset currently exists for AQA in TCM rehabilitation [ 7 ] , making it difficult to establish a standardized benchmark and hindering progress in this research area. To address these challenges, we propose the Cross-view Multimodality Enhanced Action Quality Assessment (CME-AQA) framework for Traditional Chinese Medicine rehabilitation training. The framework integrates both visual and skeletal information from multiple viewpoints to comprehensively capture participant movements and their interactions with the environment. Attention-based fusion [ 18 ] and multi-view learning [ 19 ] have been explored in prior AQA studies. CME-AQA advances TCM rehabilitation assessment through a task-specific integration of these mechanisms, together with dedicated dual-view benchmark datasets for this domain. The CME-AQA framework comprises two main components. The first is the Attention-based Visual–Pose Fusion (AVPF) module, which fuses visual and pose features through an attention mechanism to generate a unified and informative representation of motion dynamics and contextual cues. The second is the Multiscale View Alignment (MVA) training strategy, which leverages both first-person and third-person videos during training to establish multi-view awareness while allowing single-view inference during deployment. In addition, to address the lack of publicly available datasets and to establish a benchmark for this task, we created two synchronized multi-view datasets, shown in Fig. 1 : TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina). Each dataset includes expert-annotated ratings from 61 subjects, recorded simultaneously from first-person and third-person viewpoints, providing a valuable resource for future research in TCM rehabilitation assessment. With the proposed datasets, we conducted a comprehensive benchmark study comparing existing action quality assessment methods for physical therapy and validating the effectiveness of our CME-AQA framework. The experiments demonstrate that our approach achieves over 10% relative improvement in weighted F1 over the strongest competing baseline on key tasks such as Needle Depth and Quick Needle Insertion, while maintaining competitive performance across other metrics. These results highlight the capability of the CME-AQA framework to enhance TCM rehabilitation training through more precise and reliable measurement. Furthermore, our method reduced error by approximately 4% in metrological evaluations such as insertion time and manipulation frequency, demonstrating its effectiveness in both categorical and continuous assessment tasks. To further examine applicability beyond the proposed TCM tasks, we evaluated CME-AQA on a Cardiopulmonary Resuscitation (CPR) dataset [ 20 ] , where it achieved measurement performance comparable to human experts on several posture-based criteria, suggesting applicability to related structured simulated clinical skill assessments where participant motion is central to evaluation. The source code and proposed datasets are publicly available on GitHub 1 1 1 https://github.com/FrancisXZhang/cme-aqa . For transparency, we note that parts of the Methodology section adapt content from the author’s PhD thesis [ 21 ] and were also presented in the MICCAI 2025 Doctoral Consortium, with permission. Our main contributions are summarized as follows: 1. We propose CME-AQA, a cross-view multimodal action quality assessment framework for TCM rehabilitation training, and introduce two synchronized dual-view benchmarks, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina),

Read the complete text