A customized AI-based machine learning model for evaluating participants’ activities under a workshop setting Authors: Xiaoyi Shao, Hua Li, Qian Ma, Kai Jin Published: 2026 ABSTRACT AbstractHuman action recognition (HAR) in the workshop-style environment poses unique challenges due to class imbalance, fused action boundaries, and context dependence. This study proposes a multi-stage fine-tuning supervised framework for workshop-style HAR based on a 3D CNN, supplemented by an unsupervised post-assessment. A total of 208 clips (100 frames per clip) from two internal and two external datasets were used, along with 183 clips generated from 152 videos in a subset of the public UCF101 dataset for an add-on supplementary analysis. An I3D-ResNet50 (I3D_R50) backbone was used to examine a pilot model for domain adaptation and subsequently optimized for cross-context generalization. In Study 1, the I3D_R50 model with a customized 57-class prediction head was trained on the internal dataset, achieving a solid class- and cluster-level performance, however, spurious co-activations and limited separation among similar actions were observed. Study 2 refined sample allocation and data augmentation, and applied threshold tuning to evaluate within-context adaptation (Test 1) and cross-context generalization (Test 2). Multi-level analyses (class-, clip-, cluster-, and person-level) revealed moderate co-activation and semantic fusion within context, whereas cross-context generalization suffered from structure over-merging and semantic drift. As a post-assessment, Study 3 provided supplementary evidence that the model captured certain semantic co-occurrence relationships, although its performance remained constrained by limited IGE-TREE specific training samples and fine-grained behavioral categories. Limitations include insufficient samples, blurred temporal sequence, and context dependency. Future work could explore domain-adversarial alignment, pose-guided normalization, and temporal contrastive modeling, and selective merging of semantically similar actions to enhance semantic disentanglement and generalization.Keywords: AI-based Machine Learning, YOLO, I3D, Adaptation, GeneralizationSubject terms: Computational biology and bioinformatics, Engineering, Mathematics and computing Introduction Evaluating human activities in workshop-style settings has both practical values and technical challenges. Nowadays, AI-recognition techniques are used for real-time examinations and prediction. For example, some supermarkets are now using object and face recognition for auto-checkout, while highways are applying camera systems to identify speeding vehicles and prevent hit-and-run incidents. However, such techniques always come with challenges as real-time co-occurring actions (e.g., reading while note-taking), frequent occlusions, complicated manual labeling, and restricted transferability from domain into particular contexts. These problems motivate comprehensive studies focus on tailoring AI-assisted machine learning pipelines to cope with human action recognition (HAR). There are two main streams for HAR. One is image-based, and the other is spatiotemporal. Image-based action recognition focuses on extracting spatial features and performing real-time detection from single or a few video frames. This task serves as the foundation for subsequent complex spatiotemporal studies. The YOLO family1 is widely used for real-time detection and applied in tasks such as intelligent surveillance and behavior analysis2. Among YOLO series, YOLOv8 introduced an anchor-free split head for better accuracy and more efficient detection process1. YOLOv8 is also a milestone and mature version since some previous versions, such as YOLOv3 and YOLOv5, used the updated detection head from v8. YOLO is a one-stage detector pretrained on ImageNet3 that processes a target image in a single pass to obtain prediction results, including bounding box location, class name, confidence, and achieving high detection speed4–6. However, unlike prior works that treated object detection as a classification problem, YOLO formulated it as a regression problem, directly mapping image inputs to spatial bounding boxes and associated its class probabilities7. Early research demonstrated that YOLO-based HAR and localization can achieve high accuracy using only a small number of frames or even a single image, which highlighted the model’s feasibility for real-time applications8. Currently, YOLO series are applied in classroom monitors to detect students’ activities2,5,9,10. Region-based Convolutional Neural Networks (R-CNN) are another family of machine learning models for detection and localization, and they are two-stage detectors. It is a sequential approach that at Stage 1, localizing objects via region proposals, then at Stage 2 refining and classifying them. This process enables high accuracy and robustness, especially for complex scenes11,12. Other variant representative architectures are Fast R-CNN13, Faster R-CNN14, and Mask R-CNN15. Fast R-CNN shares CNN computations across images, greatly improving detection efficiency. Girshick improved training speed and testing time based on R-CNN and achieved higher mean average precision13. Ren et al. embedded RPN (Region Proposal Networks) into the network to enable end-to-end training in the spirit of attention, and unified the RPN and Fast R-CNN into a single network14. Building on Faster R-CNN, Mask R-CNN adds a branch to output pixel-level masks for each detected object, enabling efficient detection and precise instance segmentation simultaneously15. Unlike image-based methods applying static cues, spatiotemporal recognition focuses motions across sequential frames. These models learn dynamics at different perspectives. For example, 3D CNNs convolve over Frames×Height×Width to capture short-to-mid temporal patterns; RNNs/LSTMs/GRUs attach memory over visual features to encode temporal relations; and Transformer-based models use attention mechanism to capture global, long-range spatiotemporal relations. Therefore, spatiotemporal HAR models can be generally categorized into convolution-based approaches (e.g., 3D CNNs such as C3D, I3D, and SlowFast) and attention-based approaches (i.e., Transformer-based models). While 3D CNNs focus on learning local or multi-scale temporal patterns through convolutional operations, Transformer-based models explicitly analyze long-range dependencies via global attention, making them suitable for complex and context-dependent action understanding. Recent studies further extend Transformer-based architectures to fine-grained human-centered multi-label recognition tasks. For example, Bui et al. introduced that CLEAR integrates cross-Transformers with a pretrained model to jointly learn visual and semantic representations for person attribute recognition and retrieval16. Similarly, C2T-Net combines multiple Transformer-style networks with channel-aware attention and cross-fusion mechanisms to capture complementary local and global features17. These studies demonstrated the effectiveness of Transformer-based feature fusion and semantic modeling in handling complex, multi-label human behaviors. Although these models are primarily developed for attribute recognition rather than action recognition, they are highly relevant to workshop-style HAR, where behaviors are often co-occurring, visually similar, and context-dependent. Their design suggests that attention-based architectures may help alleviate challenges such as semantic fusion and over-activation. For another, 3D CNN also forms a mainstream of spatiotemporal models for video understanding. C3D was the first to show that extending 2D filters into 3D enables a single backbone to learn both appearance and short-range motion directly from fixed-length video clips18. I3D advanced this idea by “inflating” ImageNet-pretrained 2D kernels into 3D, and preserved successful architectures and initialization to boost accuracy and data efficiency19. In practice, I3D-based models are commonly pretrained with large-scale video datasets such as Kinetics-40020,21, which contain hundreds of generic human action categories and provide a strong initialization for transfer learning in downstream tasks. SlowFast further decouples semantics and dynamics with a low-frame-rate slow pathway (appearance) and a high-frame-rate fast pathway (motion), and fuses them for a strong accuracy and efficient balance22,23. Based on frame-level visual features (e.g., 2D/3D CNN), the recurrent models, such as RNNs, LSTMs, and GRUs, introduce a hidden state that carries information across time, enabling the network to capture sequence features beyond the fixed temporal window of pure CNNs24–26. In practice, sequential frames are input to a directional recurrent layer, optionally stacked, and trained by either many-to-one for clip labels or many-to-many frame labels. These models are efficient and able to handle variable-length clips well. However, computation limitation and long temporal sequences motivate the attention mechanism, such as Transformer. Recent workshop-style studies typically applied in the intelligent classroom settings, which adopt detector-centric pipelines (i.e., YOLOv7/v8/v10 variants; sometimes with pose and skeleton) to localize actors and then detect actions with either single-frame cues or a smaller setting combines detection with sequence models (e.g., I3D). Li et al., Jia and He used YOLOv5 for localization and added OpenPose skeletons for action recognition, and improved robustness under complex backgrounds27,28. Yang and Yang et al. improved YOLOv7 with bi-level routing attention, evaluated and tested it on merely eight classroom actions (e.g., hand-raising/reading/writing)29,30. Chen et al. integrated Res2Net-style modules into YOLOv8 to handle density, occlusion, and multi-scale issues in classroom videos9. Sheng et al. improved YOLOv8s and designed a Multi-scale Large Kernel Convolution Module (MLKCM) plus a Progressive Feature Optimization Module (PFOM) to adaptively capture multi-scale features for real-time detection10. Those studies focused on student actions in the classroom, but Ye et al. used HRNet skeletons to classify instructors’ actions, arguing that pose suppresses background noise. This pose-only pipeline struggled with filtering human-object interactions (e.g., reading and writing have similar pose locations)31. However, the previous studies were still image-based detection, lack of temporal association and person-level tracking construction. Given that Rafique et al. applied 3D CNN and Conv2D LSTM with time-distributed layers which were used for a range of action recognition for a classroom session, they reported results were effective for teaching technique analysis32, but only instructor-centric detection still lack of multi-person tracking. Similar to the instructor action recognition, Lu et al. developed a Erhu bowing posture training system which provided personalized and real-time feedback for the fresh Erhu learners by detecting bow and hand, computing bow-level/straightness scores, and generating instructor-ready reports4. This is a typical case study that adapted YOLO model in practice and demonstrated measurable features in accuracy and stability under AI system assistance. Recent workshop-style research relies heavily on YOLO-style CNN detectors with customized neck/heads for multi-scale issues (e.g., WAD-YOLOv833, MLKCM-YOLOv810, YOLOv8-Res2Net9 variants) and typically evaluate on SCB/FSCB datasets. On the other hand, special education (e.g. pattern recognition for children with autism34), clinical training (e.g. medical and nursing education35), and arts instruction (e.g., Erhu bowing training with YOLO4) highlighted the task- and scene-specific nature of workshop-style professional skills, showing that generic datasets are not suitable for such specialized contexts. This current study This study faces a context-specific situation, and the dataset comes from IGE-TREE workshop videos36 which content essentially differs from previous typical classroom-based studies (e.g., K-12 classroom, arts education or medical training along with image-based behaviors). The IGE-TREE workshops involve higher-education participants (Master level or above), complex workspaces, variable viewpoints, and background occlusions. Therefore, we adopted a “detection → tracking → recognition → evaluation → adaptation → generalization” pipeline to produce person-centric and ID-stable clips for training, validation, and test. An HAR model is customized for the IGE-TREE workshop actions, with the expectation that it can be extended to other transdisciplinary workshop settings. In this study, our primary objective is to investigate context-specific adaptation rather than to achieve strong cross-domain generalization. Therefore, the proposed framework is intentionally optimized for workshop-specific behavioral patterns, which may limit its transferability to other external datasets. Unlike large-scale public datasets (e.g., Kinetics-400 or other benchmark datasets), the action labels in this study are specifically designed to reflect context-dependent workshop activities. Directly adopting all the action categories from public dataset may introduce irrelevant or redundant classes that do not appear in our IGE-TREE workshop settings. Therefore, we adapt an I3D_R50 model (pretrained on Kinetics-400) to align with the task-specific behaviors, ensuring better semantic consistency between training data and real-world application scenarios. From a model selection perspective, although Transformer-based architectures have demonstrated strong capability in modeling long-range dependencies and complex semantic relationships, they typically require large-scale datasets to achieve stable and reliable performance. In contrast, 3D CNN-based models (e.g., I3D and its variants) are more data-efficient and robust under limited-sample conditions. Given the relatively small dataset size and strong domain specificity in this study, we adopt a 3D CNN-based backbone (I3D_R50) to better balance model capacity, training stability, and practical applicability. This research addresses HAR in workshop-style environments, where actions are visually similar, context-dependent, and difficult to distinguish. The proposed framework incorporates AsymmetricLoss (ASL)37 to reduce the dominance of easy negative classes and improve learning for underrepresented actions. Along with the performance evaluation across both familiar and unfamiliar settings, enabling a distinction between adaptation and generalization ability. Since this study applies a fine-grained multi-label supervised recognition framework to workshop-style behavioral analysis where multiple semantically related actions may co-occur within the same short clip, an add-on unsupervised post-assessment further supports this design by showing that threshold sensitivity analysis of related-action hit rates on a subset of public HAR dataset can reveal whether the model co-activates semantically relevant IGE-TREE behaviors beyond the internal dataset. Third, beyond reporting overall performance metrics, the study conducts a multi-level analysis of model predictions to provide deeper diagnostic insights into recognition errors and model behavior. The overall system integrates detection, tracking, and spatiotemporal recognition modules, combining YOLO-based detection with an I3D_R50 action recognition backbone to form a practical pipeline for automated behavioral analysis in workshop-style learning environments. It is important to note that this study is not designed as a performance benchmarking work across multiple action recognition architectures. Instead, model selection is guided by the characteristics of the target dataset and task setting, including limited sample size, class imbalance, and temporally continuous behavioral patterns. This design allows the study to focus on diagnosing model behavior and identifying failure patterns under domain-specific constraints, rather than comparing performance across different model families or pursuing state-of-the-art results on standardized benchmarks. Results The present study comprises three sub-studies. Study 1 built a pilot model with a customized 57-class classification heads to explore how this model understands the IGE-TREE workshop context from detection to evaluation. Study 2 refined the pilot model and used this fine-tuned model to evaluate its performance across contexts aiming to examine the model performance of adaptation (Test 1) and generalization (Test 2). Study 3 was added as a public dataset post-assessment, in which the fine-tuned 57-class IGE-TREE model was applied to a subset of a public HAR dataset (UCF10138–40) as unsupervised inputs. This add-on analysis used threshold sensitivity analysis of IGE-TREE related-action hit rates to examine whether the model could activate semantically relevant IGE-TREE workshop-style behaviors beyond the internal dataset. Study 1: Performance of the pilot tuned model A total of 208 clips from internal Dataset_1 (IGE-TREE workshop) were split into training (145 clips), validation (31 clips), and test (32 clips) sets at ratio of 70% : 15% : 15% in Study 1. We adopted the pretrained I3D_R50 model’s backbone, but replaced its original heads with 57 classes corresponding to IGE-TREE workshop actions. Performance was evaluated on the validation set after each epoch. Since this is a multi-label task and our data is imbalanced and long-tail, we will not use “Macro” measurements. Therefore, to compare 30 epochs, we ranked checkpoints primarily by Micro-F1 (low variance under class imbalance). Ties are broken by Weighted-F1 to encourage balanced Precision-Recall across labels, then by Weighted-Recall to prioritize sensitivity (fewer missed actions), and finally by Validation Loss as a proxy. The Epoch 16 was optimal in all 30 epochs. We prioritize the Epoch 16 checkpoint where Micro-F1 and Weighted-F1 peaked, and the Loss is close to its minimum. This represents a pragmatic trade-off rather than severe overfitting (Fig. 1). Figure 2 summarizes accuracies on validation set, and among them, Positive-label Accuracy, Jaccard Accuracy, and Subset Accuracy show a local peak around Epoch 16. Therefore, we selected the optimal model at Epoch 16, which is used to evaluate the test set. Epoch-level validation evaluation curves for the pilot learning model. Comprehensive view of validation accuracies for the pilot learning model. The test results were shown by an ID-level stats summary (Table 1). To characterize the model prediction tendency, we calculated the Prediction Tendency Ratio (PTR), defined as the average number of predicted positive labels (Avg_pred_pos_labels) divided by the average number of true positive labels (Avg_true_pos_labels). This ratio reflects whether the model tends to under-activate or over-activate labels. Because if the values of “Avg_pred_pos_labels” are close to the values of “Avg_true_pos_labels”, the PTR approaches to “1”, indicating a more balanced performance. Therefore, we categorized the PTR values into three ranges using 1±10% tolerance. PTR below 0.90 were considered as Conservative, those between 0.90 and 1.10 were considered as Balanced, and those above 1.10 were classified as Aggressive. The ±10% tolerance provides an interval that distinguishes under-prediction, near-balanced prediction, and over-prediction behaviors by the model while allowing for minor numerical fluctuations. Table 1ID-level performance on test evaluation for the pilot optimal model.ID_1ID_15ID_19N14512Label_acc0.9510.9370.969Pos_label_acc0.4680.6520.649Subset_acc (no-FP)0.6430.4000.417Jaccard_acc0.5360.5430.565Exact_match_acc0.2140.2000.167Micro_F10.5980.6250.696Weighted_F10.4850.6330.641Micro_P0.8290.6000.750Weighted_P0.5200.6750.657Micro_R0.4680.6520.649Weighted_R0.4680.6520.649Avg_pred_pos_labels2.5005.0002.667Avg_true_pos_labels4.4294.6003.083Label Activation Ratio*0.0440.0880.047Prediction Tendency Ratio*0.5641.0870.865Prediction Tendency Type*ConservativeBalancedConservative*Label Activation Ratio = Avg_pred_pos_labels/57 Classes; Prediction Tendency Ratio = avg_pred_labels/avg_true_labels; Conservative (<0.90), Balanced (0.90 - 1.10), and Aggressive (>1.10). ID-level performance on test evaluation for the pilot optimal model. *Label Activation Ratio = Avg_pred_pos_labels/57 Classes; Prediction Tendency Ratio = avg_pred_labels/avg_true_labels; Conservative (<0.90), Balanced (0.90 - 1.10), and Aggressive (>1.10). The Conservative (ID_1: 0.564; ID_19: 0.865) prediction tendency type (PTT) indicates that the model is under-prediction and may miss positive cases; Balanced (0.90-1.10) PTT is roughly consistent with the true label cardinality; Aggressive (>1.10) PTT implies that the model is over-prediction and increases false positives. These cutoffs are not fixed theoretically but practical guidelines for examining whether this model activation tendency was balanced in the multi-label classification task. Finally, this pilot learning model performed well on ID_15, and its PTR is 1.087 detected as balanced, which is better because the pilot model certainly learned prototypes from the original actions. Study 2: Performance of the fine-tuned model Despite the promising outcomes observed in Study 1, we further refined and optimized the pilot model to extend its applicability. Based on the findings from Study 1, Study 2 examined whether the fine-tuned model could fit across different contexts. Therefore, Test 1 and Test 2 under Study 2 were designed to comparatively evaluate the model’s adaptability and generalization across different contexts. Before applying the final model to our new domain session (internal Dataset_2), we reallocated samples and augmented the data to ensure a balanced and representative dataset for multi-label action recognition while achieving more reliable model effect. Thus, 208 clips (internal Dataset_1) were divided into training and validation subsets, ensuring that each subset contained clips with a comparable distribution of action categories. Categories that appeared only once in the entire dataset must be locked into the training set to ensure that all unique action types were learned by the model. Subsequently, a stratified sampling procedure was applied to the remaining data for each action category. In this process, the proportion of samples for each category in the training and validation sets was maintained to be consistent with its overall frequency in the dataset, ensuring that each action category appearing in the training set would also appear at least once in the validation set. Finally, we got Study 2’s training set (134 clips) and validation set (74 clips). Then, these 134 clips in the training set were horizontally flipped for data augmentation; therefore, there were a total of 268 clips in the training set. This stratified and augmentation strategy effectively prevented category omission and would improve the model generalization across all 57 action classes. Meanwhile, we replaced BCEWithLogitsLoss (BCE) with ASL. ASL is better suited to multi-label settings with class imbalance because it explicitly treats positives and negatives differently. It down-weights the vast number of easy negatives while keeping strong focus on the rarer positive instances. Concretely, ASL gives a higher penalty for negative samples than for positive samples, which effectively suppresses the overwhelming influence of easy negatives in imbalanced datasets. This mitigates “negative dominance”, improves recall on minority classes, and typically yields more stable gradients than BCE, which weights positives and negatives symmetrically and leads to bias toward predicting “no label” in imbalanced datasets. We also revised the optimal epoch selection criteria. In Study 2, models are prioritized first by val_weighted_F1, then by val_weighted_Precision, next by val_weighted_Recall, and finally by the lowest val_ASL to ensure stable optimization. This modification is more reasonable because these stats can reflect both class balance and prediction reliability. Weighted F1 serves as a comprehensive indicator of the model performance across all classes, particularly for the situation of label imbalance, while weighted precision and recall ensure neither over-prediction nor under-detection for evaluation. The inclusion of the val_ASL further ensures that the selected model not only achieves high predictive accuracy but also maintains stable optimization behavior during training. Together, this multi-measure strategy provides a more balanced and reliable criterion for identifying the most generalizable model compared with earlier micro- or average-based selections. In addition, we adjusted the prediction threshold accordingly from 0.05 to 0.95, and finally 0.45 is the optimal threshold to achieve a better performance. Through these adjustments, we got the optimal model at Epoch 20. Then, we used this final tuned model to evaluate its performance on the internal Dataset_2 for Test 1, and on the external Dataset_3 and Dataset_4 for Test 2. Table 2 is a summary of the fine-tuned model performance. According to the PTR, all test partitions and subject groups were categorized as Aggressive (>1.1) since we applied loose criterions and algorithms to make our final model more bold because previous model’s PTT was detected as conservative for ID_1 and ID_19. This aggressive prediction tendency is characterized by an increased number of false positives and the simultaneous activation of multiple action classes within a single clip. Such behavior suggests that the model becomes more sensitive to co-occurring actions, but at the cost of reduced precision. Notably, this phenomenon is not limited to a single level of analysis, but is reflected across action classes (confusion among similar categories), clusters (co-occurrence of related actions), and clips (over-prediction within short temporal segments). Table 2Test and ID-level performance after model adjustment.TestIDsTest 1Test 2Pool123444106165219N323971321012103112Wt_F10.470.410.360.470.890.590.560.130.501.000.30Mi_F10.280.140.220.280.110.260.100.120.290.130.17Wt_P0.370.320.280.370.830.500.430.110.501.000.28Mi_P0.170.080.130.170.060.150.050.100.200.070.11Wt_R0.850.720.810.851.000.850.940.170.501.000.33Mi_R0.850.720.810.851.000.850.940.170.501.000.33Exact_acc0.000.000.000.000.000.000.000.000.000.000.00Jaccard_acc0.160.090.120.160.060.150.050.060.170.070.11Subset_acc (any-hit)1.000.950.971.001.000.921.000.671.001.001.00Pos_acc0.850.720.810.851.000.850.940.170.501.000.33Label_acc0.520.660.600.520.550.810.500.830.740.750.74Avg_pred31.7220.2625.4231.7227.0012.0829.707.0015.0015.0013.50Avg_true6.312.184.046.311.502.171.604.006.001.004.50Label Activation Ratio0.560.360.450.560.470.210.520.120.260.260.24Prediction Tendency Ratio5.039.296.295.0318.005.5818.561.752.5015.003.00 Test and ID-level performance after model adjustment. To evaluate the fine-tuned model’s adaptability and generalizablity across contexts, we examined its characteristics on action classes, clusters, and clips. Test 1: Within-context adaptation Figure 3 shows that all the top-10 mis-pairing classes directed to C4 (fiddling pen), e.g., C19→C4, C26→C4, C7→C4, showing a systematic attraction toward pen-related micro-gestures. This matches our domain workshop scenes where subtle hand/head postures are frequent and visually similar. Top-10 class-level confused pairs (Test 1). The model prediction (Fig. 4) forms a large cluster (Cluster 1), a smaller Cluster 3 (C9, C16, C15, C40), and C36 is a singleton. The large Cluster 1 indicates the model tends to aggressively co-activate many actions together within the same clip. However, the true structure (Fig. 5) is modular. There are two medium-large clusters (Cluster 5 and Cluster 6), two small clusters (Cluster 4 and Cluster 3) and two singletons. Therefore, the model prediction packs C34, C4, C18, C14, C19, C35, C29, C39, C20, C7, C17, C54, C23, C26, C51, C55 into one large cluster differs from the true underlying structure. Clip-level co-activation heatmap and cluster prediction (Test 1). Clip-level co-occurrence heatmap and cluster ground truth for comparison (Test 1). Table 3 is a summary for the comparison of model prediction and ground truth. It explains why PC_1 (model prediction cluster 1) looks “too big”. MTR = 2.402, i.e., labels over-activation by about 2.4 times, because it almost merges two ground truth clusters (TC_4 and TC_6) into one, which indicate that the overlapped members, once pooled in a same prediction cluster, seemingly more frequently co-activated in predictions than they co-occur in ground truth. However, C35, C34, C39 as a ground truth cluster 4 (versus model prediction contributing them within the large Cluster 1), this model implies having a co-activation visual pattern in prediction. Table 3Weighted cluster-level statistics (Test 1: Model Prediction vs Ground Truth).Cluster NO.ClassesModel CountModel RateTrue CountTrue RateModel/True RatioModel Prediction 1C34, C4, C18, C14, C19, C35, C29, C39, C20, C7, C17, C54, C23, C26, C51, C559.6500.8484.0170.3532.402 3C9, C16, C15, C404.0001.0001.1670.2923.429 2C36NANANANANAGround Truth 5C9, C16, C18, C15, C29, C23, C7, C4, C514.3330.4487.5280.7791.737 6C19, C26, C55, C17, C54, C14, C204.4760.39711.2861.0002.521 4C35, C34, C394.6670.4838.1670.8451.750 3C1, C531.0000.6670.0000.0000.000 1C36NANANANANA 2C40NANANANANA Weighted cluster-level statistics (Test 1: Model Prediction vs Ground Truth). Test 2: Cross-context generalization Test 2 replicated the processing pipeline used for Test 1 for examining the fine-tuned model’s cross-context effect. Unfortunately, the model failed to correctly identify any class (the top-k correct classes = 0). This indicates that the final tuned model could no longer generalize class-specific visual cues effectively when applied to completely unfamiliar workshop scenes. The lack of correct predictions suggests that previously learned action distinctions, such as between hand or pen motions, were overwhelmed by contextual shifts. For the top-10 confused pairs, the confusions (Fig. 6) also converge around C4, but unlike Test 1 (Fig. 3), the misclassifications C23→C4 and C6→C4 dominate in Test 2, showing the model prediction a strong bias toward pen- and hand-related gestures when tested in an unfamiliar workshop-style scene. This model suggests that visual similarity between “holding pen” and “fiddling pen” across contexts, even though the environmental layout, lighting, or person differ. The model also predicted “flipping pages” as “fiddling pen” likely because both actions take place on the participant’s workspace and exhibit similarly rapid hand motions, making them difficult for the model to distinguish. Top-10 class-level confused pairs (Test 2). Table 4 is also a summary for the comparison of model prediction and ground truth. The giant Cluster 2 in model prediction shows an MTR of 7.952, indicating that predicted co-activations occur about 8 times more often than their true frequency, which is an extreme case of over-generalization. Meanwhile, the small pair C35, C46 yields MTR 0.250, mirroring its true cluster TC_7 (MTR = 0.206), showing under-occurrence for this fine-grained group. Among true clusters, the MTR of TC_8 is 1.250 closest to 1, implying that the model correctly captured their co-occurrence frequency. This outcome supports that “turning head” (C54) and “turning back” (C55) occur together in real interactions. Table 4Weighted cluster-level statistics (Test 2: Model Prediction vs Ground Truth).Cluster NO.ClassesModel CountModel RateTrue CountTrue RateModel/True RatioModel Prediction 2C15, C39, C33, C16, C17, C26, C55, C54, C57, C23, C434.4820.7830.5640.0987.952 1C35, C460.5000.1672.0000.6670.250Ground Truth 7C35, C50, C6, C462.8330.7080.5830.1460.206 9C26, C42, C1, C72.0000.8890.5000.2220.250 5C23, C578.0000.42117.5000.9212.188 6C17, C331.0000.6670.5000.3330.500 8C54, C552.0000.8002.5001.0001.250 1C15NANANANANA 2C39NANANANANA 3C16NANANANANA 4C43NANANANANA Weighted cluster-level statistics (Test 2: Model Prediction vs Ground Truth). Study 3: Post-assessment on public HAR dataset for related-action sensitivity To further examine the generalizability of the optimal model beyond the original IGE-TREE and small Youtube dataset, we conducted a public dataset post-assessment using the “WritingOnBoard” (WoB) category, total 152 videos from UCF10138, a public HAR dataset. This category was selected because it was the closest publicly available action category to the IGE-TREE workshop context. Although the public dataset does not provide the same fine-grained behavioral labels as the IGE-TREE dataset, WoB videos commonly involve IGE-TREE related behaviors, such as subtle body movement and hand/head actions that are semantically relevant to our model learning environment. The public WoB data were processed to match the input format of the trained I3D-R50 model. In total, 152 public videos were converted into 183 video clips. Each clip contained 100 frames, consistent with the input length used in the original model training pipeline. Tail segments with fewer than 100 frames were discarded to maintain a fixed-length input structure. The public dataset provides only a general label “WritingOnBoard”, however, the videos contain multiple fine-grained behaviors which actually co-occur in the WoB and are relevant to the IGE-TREE setting. There are at least nine IGE-TREE related actions, including “walking, turning body, holding pen, looking sideway, hand reaching out, turning head, hand up, hand down, and writing”. Therefore, WoB categorgy was selected to have a post-assessment. The optimal 57-class IGE-TREE model was applied to the 183 public WoB clips as an unsupervised post-assessment. For each clip, the model generated a 57-dimensional probability vector. A clip was counted as a related-action hit (hit_acc) if at least one of the nine IGE-TREE related labels exceeded a given probability threshold. Therefore, the reported metric should be interpreted as an IGE-TREE related action hit rate rather than conventional classification accuracy. The threshold sensitivity analysis (Fig. 7) showed that the model achieved an IGE-TREE related action hit rate of 82.5% at threshold 0.45 and 77.6% at threshold 0.50. This medium-high result provides a post-assessment indication that our model can activate semantically relevant IGE-TREE actions on the public HAR dataset, even though the public dataset does not provide full fine-grained annotations. Threshold sensitivity: related hit count and hit accuracy. Discussion Our findings show that this customized model performs substantially better on the internal dataset than on the external dataset, indicating limited generalization ability. It should be noted that the external evaluation is based on a relatively small number of clips, which limits the statistical strength of the results. Although each clip contains rich temporal information (100 frames per clip), the evaluation is conducted at the clip level, and thus the effective sample size remains limited. Therefore, the reported generalization performance should be interpreted as exploratory and indicative rather than conclusive. It should also be noted that model performance is influenced by hyperparameter configurations, which introduce inherent trade-offs between convergence stability and predictive behavior. These trade-offs should be interpreted as part of the model behavior rather than definitive performance characteristics. Consistent with the study’s objective, this work is not intended to establish strong cross-domain generalization claims, but rather to provide a diagnostic analysis of model behavior under data-limited and cross-domain conditions. More specifically, the limitations manifest across four levels of analysis. At the class-level, the model struggles with fine-grained distinctions between visually similar actions. It should be noted that the current 57-class label design is intentionally preserved to support another study for analyzing workshop engagement in the IGE-TREE program. Although some visually similar actions could be merged, they may correspond to different engagement states (e.g., disengagement vs. engagement), and merging them would reduce interpretability in subsequent analysis. At the clip-level, temporal boundaries remain unclear and the number of training samples is insufficient, which constrains temporal modeling. At the cluster-level, actions that frequently co-occur in real settings tend to be overly co-activated in the model’s predictions, leading to inflated cluster activations. At the person-level, the dataset lacks diversity in participants, and the predictions are further influenced by the spatial relationship between the participant and the camera view angles. These observations reflect its sensitivity to co-occurring actions and contextual variations. Importantly, the performance degradation is class-dependent rather than uniform, and it is not caused by label inconsistency, as both internal and external datasets follow the same annotation protocol. The post-assessment of threshold sensitivity also supports the rationale for the clip-level co-occurrence/co-activation analysis. Workshop-style behaviors are often not represented by a single isolated action, but by multiple semantically related fine-grained activations within the same short video clip. Limitations and future expectation Despite the above findings, several limitations remain. First, the overall dataset size is relatively small, and the number of training clips is insufficient for robust temporal modeling. Temporal boundaries within clips are not explicitly annotated, which limits the model’s ability to learn precise temporal structures. In addition, the dataset lacks diversity in participants and viewpoints, which further affects generalization performance. In addition, although basic data augmentation strategies were applied during training, their impact on cross-context generalization remains limited. This indicates that the performance gap cannot be fully addressed by increasing intra-domain variability alone, but is more fundamentally related to domain shift and semantic differences across datasets. Given the current dataset scale and annotation characteristics, advanced techniques such as domain adaptation (e.g., domain adversarial training) are not explored in this study and are left as future work. These approaches may help reduce domain discrepancies and improve generalization across different contexts. From a model architecture perspective, this study is designed as a diagnostic analysis under domain-specific constraints, rather than a performance benchmarking across different model architectures, which represent a promising direction for future research. Transformer-based architectures provide strong capability for modeling global dependencies, while advanced 3D CNN variants (e.g., SlowFast) offer improved multi-scale temporal modeling. However, due to the limited dataset size and strong domain specificity, such models are not explored in the current study. A natural extension of the present framework is to adopt more advanced 3D CNN variants such as SlowFast, which can better capture multi-scale temporal dynamics by separating semantic and motion pathways. This type of improvement remains consistent with the current convolution-based design and can be considered as an upgrade version under limited data conditions. Therefore, future studies may incorporate both Transformer-based approaches and upgraded 3D CNN variants as part of a broader benchmarking framework to systematically evaluate their effectiveness in IGE-TREE workshop-style HAR scenarios. Future work will focus on expanding the dataset as more workshop sessions become available, and may explore hierarchical label structures, consider merging semantically similar actions, and incorporate segment-level annotations to improve temporal modeling. More robust evaluation strategies, such as cross-validation (e.g., leave-one-workshop-out), may also be adopted to better assess generalization across different workshop settings. In addition, future studies may incorporate interpretability and reliability analysis techniques (e.g., attention-based visualization, feature attribution, and confidence calibration) to better understand model decision patterns and mitigate overconfident predictions in cross-context scenarios. Methods Toolkits Anaconda Navigator 2.6.6, Spyder 6.0.7, and Visual Studio Code 1.121.0 for data analysis with Python 3.12.3 as programming language. Excel of Word 365 package is used for data saving and construction. Model pipeline The proposed system follows a multi-stage pipeline including person detection, tracking, clip extraction, and action recognition. Figure 8 is a brief diagram to visualize the data flow and the customized model structure. Customized I3D_ResNet50 architecture for 57 multi-label actions recognition. YOLOv8 is used to detect human subjects in each frame. The detected bounding boxes are then associated across frames to track persons’ trajectories (DeepSORT). Based on these trajectories, video clips are constructed and used as inputs for spatiotemporal action recognition using the I3D_R50 model. YOLOv8 was selected as the detection technique due to its stable performance and wide adoption in real-time object detection tasks33,41,42. Although newer YOLO variants have been proposed, previous studies indicate that performance differences among recent YOLO versions are often application-dependent and relatively limited for standard detection tasks43. Therefore, YOLOv8 provides a reliable and efficient base for the detection stage without affecting the overall performance of the action recognition pipeline. Specifically, the constructed clips (100 frame per clip) are fed into a fine-tuned I3D_ResNet50 network, where spatiotemporal features are extracted through a 3D convolutional backbone and mapped to multi-label predictions via a customized classification head with sigmoid activation. This design enables the model to capture both spatial and temporal dynamics of human behaviors within each clip. In this pipeline, the model is selected and adapted to align with the characteristics of the IGE-TREE workshop dataset, rather than to compare different model architectures. Data collection Data source The dataset consists of internal, external, and public sources. The internal data are two videos, Video_1 (15‘17“) and Video_2 (2‘40“), from two internal IGE-TREE workshops. This is a National Science Foundation (NSF) funded Innovation in Graduate Education project named Transdisciplinary Research in Graduate Engineering Education (IGE-TREE, Project Award No. 2225109). The IGE-TREE project will have a series of workshop training sessions with a transdisciplinary topic which may cover the field of technology, engineering, environment, and humanity, etc.. The external data are also two workshop-style videos, Video_3 (2‘30“) and Video_4 (1’32”), downloaded from YouTube44,45. The public dataset comes from a subset of UCF10138, the “WritingOnBoard” action category. This subset contained 152 videos, with an average duration of 6.40 seconds, ranging from 2.28 to 12.84 seconds. Samples and participants Our samples are clip-level. Internal Video_1 clips (208) forms Dataset_1, which is used for training (134 clips before flipped) and validation (74 clips). The external Video_3 clips (randomly selected 32 clips from 144 clips) and finally seven Video_4 clips were kept after removing those with missing detections or with inconsistent identity assignments across frames. Participant demographic information was not required. Only the participants in the internal videos are considered human participants in the study. The internal participants (ID_1, ID_15, and ID_19) are graduate students. Informed consent was obtained from all participants after being fully informed about the study prior to their participation. The ethics committee of the authors’ affiliation approved this research, and all research was performed in accordance with approved guidelines/regulations. No direct participants from the external and public site. No consent or demographic considerations apply to the external video subjects. The external participants (ID_2, ID_3, and ID_4 from Video_3; ID_44, ID_106, ID_165, and ID_219 from Video_4) come from two YouTube videos44,45. This research involves no more than minimal risk. This is not a biomedical/clinical trial or intervention, and the study is not intended to change participant behavior. Table 5 is a summary of data sources describing their application in this study. Table 6 presents the person-camera relationships, along with the qualitative distance classification (near, medium, or far) between participants and cameras. No person-camera information was provided by the public WoB dataset. Table 5Data source summary.Data SourceVideoDatasetIDsClipsApplication (clips)InternalVideo_1Dataset_1ID_1, ID_15, ID_19208Study 1: Training (145)Study 1: Validation (31)Study 1: Testing (32)Study 2: Training (134)Study 2: Validation (74)Study 2: Testing (32)Video_2Dataset_2ID_132Study 2: Test 1ExternalVideo_3Dataset_3ID_2, ID_3, ID_432Study 2: Test 2Video_4Dataset_4ID_44, ID_106,ID_165, ID_2197Public152 VideosUCF101_WoB–183Study 3: Post-assessment Data source summary. ID_44, ID_106, ID_165, ID_219 Table 6Participant - Camera position relationships.Subject_IDCameraViewBody OrientationDistanceVideoClarity*Remarks(relative to camera)ID_1FrontRight profileNearLow–(body turned )ID_15Body facing front–(body frontal view)ID_19Left profilePartial facial visibility**(body turned )ID_1FrontBody facing frontMediumLow–(body frontal view)ID_2FrontBack facingNearClear–(body turned )ID_3Left profileHead almost turned away;(body turned )Almost back facingID_4Right profileHead almost turned away;(body turned )Almost back facingID_44MovingcameraLeft profile/Body facing frontFarLow–ID_106Body facing frontNearClear–(head turned right )ID_165Body facing front–(head turned left )ID_219Body facing front–(head turned right )*The scale of video clarity is subjectively described by the author. **ID_19 is partial facial visibility as the beard on the chin occludes key facial features. The external participant-camera observation data are subjectively described by the author, as the original video sources did not provide such information. No person-camera information was provided by the public WoB dataset. Participant - Camera position relationships. Moving camera *The scale of video clarity is subjectively described by the author. **ID_19 is partial facial visibility as the beard on the chin occludes key facial features. The external participant-camera observation data are subjectively described by the author, as the original video sources did not provide such information. No person-camera information was provided by the public WoB dataset. Data cleaning and labeling After removing invalid frames or clips, 279 clips were standardized to 100 frames per clip (25 FPS). Then we labelled these clips manually. The same preprocessing operations on the public WoB dataset (183 clips) with unsupervision for a post-assessment. A brief description was provided in Table 5. To ensure annotation consistency, all clips (both internal and external) were annotated using the same 57-class label structure. The annotation process followed a unified protocol that each clip was reviewed frame-by-frame, and one or multiple action labels were assigned based on observable behaviors within the clip. For the external dataset, videos were carefully selected to be closely aligned with the IGE-TREE workshop setting, and all clips were annotated strictly following the same criteria as the internal workshop data. This design ensures that no label mismatch between datasets, and that all evaluations are conducted within a consistent action space. Actions that did not correspond to any of the 57 categories were excluded from the dataset to maintain label consistency. Since it is a multi-labelling task, each clip will have one or more labels. For example, in our internal workshop episodes, “holding pen” was the most frequently observed action, accounting for 19.05% of the total counts of labels (1071), whereas “grabbing files” occurred only once (0.09%) out of the total instances. Figure 9 shows the frequencies and cumulative proportion of each action class. Our data are classes imbalanced, and all 57 classes exhibit a clear “long-tail” distribution. Class frequency pareto (total counts = 1071). Data imbalance mitigation To mitigate the class imbalance, we adopted two complementary strategies. First, during dataset partitioning, we ensured that all action categories were included in the training set. Rare classes with extremely limited samples, e.g. for a sample which categories appeared only once, were explicitly assigned to the training set so that the model could learn representations for every category. Second, in the model training, we adopted ASL instead of BCE. ASL is designed for imbalanced multi-label classification, where negative labels usually far over number of positive labels. In our implementation, the negative term is down-weighted using an asymmetric focusing factor, while probability clipping was applied to suppress the dominance of easy negative samples. This design reduces majority-class bias and encourages the model to focus more on informative positive labels. Data transform and augmentation Images were transformed to PyTorch tensors (pixel values scaled to [0,1]) and standardized per channel with mean 0.45 and standard deviation 0.225, following the Kinetics-style normalization used by the pretrained I3D model. We applied basic data augmentation strategies for expanding training samples and improve model robustness from both temporal and spatial perspectives. For temporal augmentation, a sliding-window strategy was adopted to generate training clips. Each clip contains a fixed length of 100 frames, with a stride of 90 frames between consecutive clips (i.e., 10-frame overlap). This design introduces temporal variability by allowing adjacent clips to share partial motion information while still capturing slightly different temporal contexts, thereby increasing sample diversity. For spatial augmentation, horizontal flipping was applied to training samples. This operation increases spatial diversity and helps the model become more robust to variations in viewpoint and participant orientation, which are common in real-world workshop environments. Analytical procedures We used pretrained machine learning models during training and validation as our pilot Study 1 and we used fine-tuned model in the test adaptation and generalization process for Study 2. All the parameters and measures were set at the start to secure the results of the datasets from different sources can be compared. In this current study, total four AI-based machine learning models were adopted and they are YOLOv8n, DeepSORT, I3D-ResNet50 pretrained on Kinetics-400, and I3D-ResNet50 fine-tuned class-57 model. For the procedure, we trained for 30 epochs with a batch size of 3. After each epoch, we evaluated them on the validation set, and selected an optimal one under the criterion order by weighted_F1 first, then by weighted_Precision, next by weighted_Recall, and last by the smaller ASL. We also modified the loss function by AsymmetricLoss, a variant designed to better handle class imbalance in multi-label classification tasks. The final decision threshold was set to 0.45 to encourage more decisive model predictions. In Study 3, threshold sensitivity analysis was further used as a post-assessment, where related-action hit-accuracy was adopted to evaluate whether the final model captured semantically relevant IGE-TREE behaviors. Acknowledgements Dallas ISD (Youtube account @DallasSchoolsTV) granted permission to use Video_444 and the author of Video_3 enabled its public accessibility45. UCF101 provided the official website to download the datasets38. Author contributions X.S. conducted this study and drafted the manuscript. H.L., the principal investigator of the IGE-TREE project, supervised this study. All authors reviewed and revised the final manuscript. Funding This research is based on work supported by the National Science Foundation under Award No. 2225109. Data availability Since the only foreseeable risk is a potential breach of confidentiality, access to the internal-site data is restricted and granted only upon request. The external-site and UCF101 videos are publicly available, with links provided in References38,44,45. Declarations Competing Interests There is no conflict of interest among the internal sites. Ethical approval Texas A&M University-Kingsville Institutional Review Board (IRB) approved the IRB protocol 2022-039/12559. The authors have followed the approved procedures when handling the data. Informed consent was obtained from all participants after being fully informed about the study prior to their participation. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement Since the only foreseeable risk is a potential breach of confidentiality, access to the internal-site data is restricted and granted only upon request. The external-site and UCF101 videos are publicly available, with links provided in References38,44,45.