Automatic Temporal Segmentation for Post-Stroke Rehabilitation: A Keypoint Detection and Temporal Segmentation Approach for Small Datasets

Jisoo Lee, Tamim Ahmed, Thanassis Rikakis, Pavan Turaga · 2025

Read Automatic Temporal Segmentation for Post-Stroke Rehabilitation: A Keypoint Detection and Temporal Segmentation Approach for Small Datasets by Jisoo Lee, Tamim Ahmed, Thanassis Rikakis, Pavan Turaga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19766.txt.

Introduction

Lee and co-authors present in 2025 a tool that splits rehab video into time segments for stroke patients. Rehab is vital after stroke, which mostly strikes adults over 65, but scoring by eye is slow and varies by judge. The tool aims for steady, fast scoring from video alone.

It first tracks body points in each frame, then cuts the point track into phases over time. The test case is tabletop object use, a core drill in arm rehab. The design leans on how bodies move in therapy and learns from small datasets. Such a tool could help therapists adjust care sooner and track gains with less hand work.

Catalog record

CREATORJisoo Lee, Tamim Ahmed, Thanassis Rikakis, Pavan Turaga
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19766

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Temporal Segmentation for Post-Stroke Rehabilitation: A Keypoint Detection and Temporal Segmentation Approach for Small Datasets Jisoo Lee, Tamim Ahmed, Thanassis Rikakis, Pavan Turaga Abstract Rehabilitation is essential and critical for post-stroke patients, addressing both physical and cognitive aspects. Stroke predominantly affects older adults, with 75% of cases occurring in individuals aged 65 and older, underscoring the urgent need for tailored rehabilitation strategies in aging populations. Despite the critical role therapists play in evaluating rehabilitation progress and ensuring the effectiveness of treatment, current assessment methods can often be subjective, inconsistent, and time-consuming, leading to delays in adjusting therapy protocols. This study aims to address these challenges by providing a solution for consistent and timely analysis. Specifically, we perform temporal segmentation of video recordings to capture detailed activities during stroke patients’ rehabilitation. The main application scenario motivating this study is the clinical assessment of daily tabletop object interactions, which are crucial for post-stroke physical rehabilitation. To achieve this, we present a framework that leverages the biomechanics of movement during therapy sessions. Our solution divides the process into two main tasks: 2D keypoint detection to track patients’ physical movements, and 1D time-series temporal segmentation to analyze these movements over time. This dual approach enables automated labeling with only a limited set of real-world data, addressing the challenges of variability in patient movements and limited dataset availability. By tackling these issues, our method shows strong potential for practical deployment in physical therapy settings, enhancing the speed and accuracy of rehabilitation assessments. 1 Introduction Stroke is a medical condition that occurs when the blood supply to the brain is interrupted, resulting in brain tissue damage, which can lead to disability or even death. Aging is a major contributor to stroke, with the risk doubling every decade after the age of 55, and approximately 75% of stroke patients being 65 or older [ 24 ] . Rehabilitation therapy is essential for minimizing disability and helping patients recover physical and cognitive functions. This need is especially critical for older adults, as aging leads to greater vulnerability in both physical and cognitive functions, making rehabilitation even more necessary. Effective rehabilitation relies on thorough assessments to tailor treatment plans. Currently, therapists monitor these assessments, evaluating the patient’s physical and cognitive progress to refine therapy protocols accordingly. However, this process is time-intensive and may vary among therapists, creating a need for more objective, automated solutions. Advancements in deep learning have enabled automation across various fields, including healthcare. In rehabilitation, deep learning offers the potential to streamline and enhance the assessment process. Despite this potential, challenges remain, including the limited availability of real-world patient data, difficulties in using synthetic data, and the computational demands of processing video data, which often involve spatial and temporal complexities. To overcome these challenges, we propose a novel framework that decomposes complex tasks into smaller, more manageable sub-tasks based on domain-specific insights. By focusing on critical hand-object interactions during the Action Research Arm Test (ARAT), we isolate key movements and extract 2D joint coordinates for detailed analysis. This targeted approach allows us to reduce model complexity and mitigate overfitting, even with a small dataset. We utilize the ASAR (Affective State for ARAT Rehab) dataset [ 2 ] , which consists of video recordings of stroke patients performing the standardized Action Research Arm Test (ARAT) [ 25 ] . ARAT assesses upper extremity motor function in stroke patients by evaluating their ability to perform tasks such as grasping, moving, and lifting objects, aiding in tracking rehabilitation progress. Additionally, given the subjective nature of the ARAT assessment, the criteria for segmenting actions may vary depending on the therapist. To address this, instead of retraining the entire model, we propose adjusting only the temporal segmentation phase to accommodate the changed criteria. This approach offers significant flexibility by allowing adaptation to different segmentation standards without the need for complete model retraining. 1.1 Small Data Statement This study leverages video recordings of 106 stroke patients performing 19 different tasks as part of the Action Research Arm Test (ARAT) assessment. The study involved training a segmentation model to identify specific timestamps in videos captured from three different perspectives: top, left, and right views. After excluding 7 tasks without video recordings and considering cases with excessive occlusion that could not be used for training, the final dataset consisted of 561, 643, and 319 data points for the top, ipsilateral, and contralateral views, respectively. Due to the real-world nature of the data, involving actual stroke patients, augmenting the dataset with time-based scaling or other data augmentation techniques could significantly affect critical aspects such as task completion time, which are important for accurate assessment. Therefore, no augmentation techniques were applied. Additionally, given the focus on capturing subtle hand movements of stroke patients, the use of synthetic data was also deemed inappropriate for this study. Due to the limitations in data size and the need for realistic, patient-specific details, this study is classified as small data research. In this study, the primary challenge is the small amount of data, in addition to the fact that the data consists of videos, which have high dimensionality. Moreover, due to the nature of the ARAT, there is a need to capture the relationship between the patient’s fingers and the target object’s movement within a very narrow range, which presents further difficulties. Due to the small dataset size, even when utilizing pre-trained models, there are significant challenges, as the characteristics of the video data differ from those of existing datasets. Furthermore, applying complex models with a large number of parameters, such as a 3D CNN [ 21 ] , to such a limited dataset can lead to issues like overfitting. To address dataset limitations and resource constraints, we propose a new framework that breaks down a large task into smaller, more manageable sub-tasks, based on a detailed understanding of the collected data. Our prior knowledge includes the fact that the patient always interacts with a single object and is consistently instructed to move the object from the table to a shelf above. Therefore, it is more efficient to focus solely on this movement, rather than the entire video frame, to estimate the timestamp of the patient’s action label. By concentrating on the target objects and the patients’ hand movements, we extract the 2D coordinates of key body joints from each video. We refine these results and apply a transformer-based model [ 22 ] for time-series segmentation to predict the action segments. This approach mitigates the risk of overfitting associated with small datasets and demonstrates effective action segmentation using limited computational resources, highlighting its potential for real-world deployment. 2 Related Work Although various studies have introduced methods for video action segmentation [ 7 , 23 , 17 , 8 ] , these methods typically rely on direct use of video data, often from extensive benchmark datasets. Additionally, since these methods are designed to infer the final labels directly from the video, they face challenges in providing intermediate process information to therapists. Furthermore, if the criteria for segmentation change, the entire dataset must be retrained, posing a significant drawback. Moreover, the datasets used in these studies typically involve labeling based on the overall assessment of human actions, rather than distinguishing fine

Read the complete text