VIFSS: View-Invariant and Figure Skating-Specific Pose Representation Learning for Temporal Action Segmentation

Ryota Tanaka, Tomohiro Suzuki, Keisuke Fujii · 2025

Read VIFSS: View-Invariant and Figure Skating-Specific Pose Representation Learning for Temporal Action Segmentation by Ryota Tanaka, Tomohiro Suzuki, Keisuke Fujii (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10281.txt.

Introduction

Ryota Tanaka and co-authors learn pose features for figure skating jump analysis in this 2025 paper. Judging a jump's type and timing takes expert eyes. Automating that call could make scoring fairer and feedback faster.

Two problems block past methods: too little labeled data, and flat views of a three-dimensional sport. Their system, VIFSS, learns poses that stay stable across camera angles. It is tuned to the specific moves of figure skating. Better pose features feed the task of splitting video into action segments. Coaches could use such tools to review landings and rotations. The work shows how sport-specific learning beats generic pose tools. Federations may one day use the scores as a shared check.

Catalog record

CREATORRyota Tanaka, Tomohiro Suzuki, Keisuke Fujii
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VIFSS: View-Invariant and Figure Skating-Specific Pose Representation Learning for Temporal Action Segmentation Ryota Tanaka, Tomohiro Suzuki, Keisuke Fujii Abstract Understanding human actions from videos plays a critical role across various domains, including sports analytics. In figure skating, accurately recognizing the type and timing of jumps a skater performs is essential for objective performance evaluation. However, this task typically requires expert-level knowledge due to the fine-grained and complex nature of jump procedures. While recent approaches have attempted to automate this task using Temporal Action Segmentation (TAS), there are two major limitations to TAS for figure skating: the annotated data is insufficient, and existing methods do not account for the inherent three-dimensional aspects and procedural structure of jump actions. In this work, we propose a new TAS framework for figure skating jumps that explicitly incorporates both the three-dimensional nature and the semantic procedure of jump movements. First, we propose a novel View-Invariant, Figure Skating-Specific pose representation learning approach (VIFSS) that combines contrastive learning as pre-training and action classification as fine-tuning. For view-invariant contrastive pre-training, we construct FS-Jump3D, the first publicly available 3D pose dataset specialized for figure skating jumps. Second, we introduce a fine-grained annotation scheme that marks the “entry (preparation)” and “landing” phases, enabling TAS models to learn the procedural structure of jumps. Extensive experiments demonstrate the effectiveness of our framework. Our method achieves over 92% F1@50 on element-level TAS, which requires recognizing both jump types and rotation levels. Furthermore, we show that view-invariant contrastive pre-training is particularly effective when fine-tuning data is limited, highlighting the practicality of our approach in real-world scenarios. † † graphicalabstract: † † highlights: We introduce a new framework for temporal action segmentation using VIFSS (View-Invariant and Figure Skating-Specific) pose representations. We construct FS-Jump3D, the first publicly available 3D pose dataset for figure skating jumps. We propose a fine-grained annotation strategy to capture precise jump procedures. Our VIFSS approach achieves superior performance on figure skating TAS tasks over 2D pose and 3D pose features. 1 Introduction Understanding human actions from videos is a fundamental task with wide-ranging applications, from sports ( Suzuki et al. 2024 ; Tanaka et al. 2023a ) to autonomous driving ( Hashimoto et al. 2015 ) , surveillance ( Dave et al. 2022 ) , and elderly care ( Kim et al. 2022 ) . In the sports domain, video data plays a critical role in applications such as replay judgment ( Spitz et al. 2021 ) , automatic judgment ( Suzuki et al. 2024 ; Tanaka et al. 2023a ) , and providing feedback during training ( Mateus et al. 2025 ) . In particular, figure skating presents a unique challenge due to the increasing technical complexity of jump elements, making video utilization indispensable. Currently, figure skating judging relies heavily on manual annotation by technical specialists and replay operators, who identify and record jump types and timings during performances, respectively. This process requires expert knowledge and a non-negligible amount of time and effort. To address these challenges, this study proposes an automated pipeline system that recognizes the type and timing of figure skating jumps from broadcast videos by Temporal Action Segmentation (TAS). TAS ( Ding et al. 2022 ) aims to segment untrimmed video into frame-wise action labels. Existing benchmark datasets for TAS mainly consist of video recordings of procedural activities, such as cooking ( Kuehne et al. 2014 ; Stein and McKenna 2013 ; Damen et al. 2022 ; Fathi et al. 2011 ) , or assembling furniture or toys ( Bansal et al. 2022 ; Sener et al. 2022 ; Ben-Shabat et al. 2021 ; Ragusa et al. 2021 ) . These datasets annotate fine-grained actions at the frame level, e.g., “take cup” and “open fridge”, and TAS models are trained to understand these procedural transitions. A widely used and successful approach divides the TAS task into two stages: extracting temporal features and segmenting the actions based on those features ( Lu and Elhamifar 2024 ; Farha and Gall 2019 ; Li et al. 2023 ) . The quality of the extracted features greatly influences segmentation performance, making feature selection a key consideration. In standard TAS tasks, visual cues from objects or tools often help recognize actions. Consequently, image-based features such as I3D ( Carreira and Zisserman 2017 ) , extracted using 3D-CNNs, are frequently adopted. However, previous studies on understanding figure skating actions ( Liu et al. 2021 ; Hong et al. 2021 ) have suggested that 2D poses or embedded pose representations may be more informative than image-based features. This reflects the nature of figure skating as a motion-centric task, where recognizing human movement is more important than the appearance of objects or backgrounds. Nevertheless, figure skating actions are inherently three-dimensional and captured from a variety of viewpoints in broadcast footage. Leveraging 3D pose information to construct view-invariant pose representations offers promising potential for this task. However, no prior work has examined the effectiveness of 3D pose-based representations for TAS. Furthermore, existing TAS studies on figure skating ( Liu et al. 2021 ; Gan et al. 2024 ) , as illustrated in Figure 1 (a), do not account for entry (preparation) or landing phases in the annotation of jumps. Since TAS models are designed to learn sequential action procedures, explicitly labeling the full jump procedure, including the transition from entry to landing, may contribute to effective learning. Figure 1 : Overview of the proposed TAS framework. (a) We propose a novel annotation strategy that explicitly models the sequential structure of jump procedures. (b-1) To capture the three-dimensional and domain-specific characteristics of figure skating jumps, we construct FS-Jump3D, the first open 3D pose dataset dedicated to figure skating. (b-2) We introduce a two-stage pose representation learning framework, consisting of view-invariant contrastive pre-training and domain-specific fine-tuning via jump action classification. Finally, we evaluate the effectiveness of the learned pose embeddings as input to TAS models, combined with the proposed annotation scheme. In this paper, we propose a new TAS framework that utilizes a View-Invariant and Figure Skating-Specific (VIFSS) pose representation learning method, which captures the three-dimensional nature of figure skating movements, as illustrated in Figure 1 . First, this study addresses the above issues by proposing a new annotation strategy for figure skating TAS tasks that incorporates detailed jump procedures. We also investigate the utility of view-invariant and figure skating-specific 3D pose representations as input for the TAS model. To address the challenges of incorporating the three-dimensional nature of figure skating motions into TAS, we introduce VIFSS that utilizes pose embedding features instead of directly using estimated 2D or 3D joint coordinates. While raw joint coordinates can represent motion geometrically, they often lack robustness to viewpoint variations and may hinder generalization across diverse domains. To overcome these limitations, we propose a two-stage learning approach. First, we pre-train a pose encoder using contrastive learning on 3D pose datasets to acquire view-invariant representations. Then, we fine-tune the encoder via action classification tailored to the figure skating domain. By combining view-invariant pre-training with domain-specific fine-tuning, our method enables more effective and flexible representation learning for TAS in figure skating, particularly under conditions of limited annotation and diverse viewpoints. Although contrastive learning for pose representation has the advantage of not requiring explicit 3D

Read the complete text