High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions
Yongpeng Caoand Yuji Yamakawa{cao, y-ymkw}@iis.u-tokyo.ac.jp · 2026
Read High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions by Yongpeng Caoand Yuji Yamakawa{cao, y-ymkw}@iis.u-tokyo.ac.jp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00496.txt.
Introduction
Yongpeng Cao and Yuji Yamakawa test whether faster video makes unfamiliar human actions easier for AI to understand without any task-specific training. Zero-shot understanding matters where actions are rapid or rare, since labeled examples are scarce. Their case study is kendo, the Japanese fencing art, whose strikes are quick and subtle.
Their training-free pipeline pairs a pretrained video-language model, which supplies semantic representations, with language-model reasoning that compares actions pairwise. The same scenes were tested at three frame rates, from thirty to one hundred twenty frames per second. Higher frame rates clearly improved how well action meanings separated in the zero-shot setting. Adding tracked body-joint information helped under both full and partial views. The authors conclude that high-speed perception yields steadier, more interpretable representations of fast action.
Catalog record
| CREATOR | Yongpeng Caoand Yuji Yamakawa{cao, y-ymkw}@iis.u-tokyo.ac.jp |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00496
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-Speed Vision Improves Zero-Shot Semantic Understanding of Human Actions Yongpeng Caoand Yuji Yamakawa{cao, y-ymkw}@iis.u-tokyo.ac.jp Abstract Understanding human actions from visual observations is essential for human–robot interaction, particularly when semantic interpretation of unfamiliar or hard-to-annotate actions is required. In scenarios such as rapid and less common activities, collecting sufficient labeled data for supervised learning is challenging, making zero-shot approaches a practical alternative for semantic understanding without task-specific training. While recent advances in large-scale pretrained models enable such zero-shot reasoning, the impact of temporal resolution, especially for rapid and fine-grained motions, remains underexplored. In this study, we investigate how temporal resolution affects zero-shot semantic understanding of high-speed human actions. Using kendo as a representative case of rapid and subtle motion patterns, we propose a training-free pipeline that combines a pre-trained video-language model for semantic representation with large language model-based reasoning for pairwise action comparison. Through controlled experiments across multiple frame rates (120 Hz, 60 Hz, and 30 Hz), we show that higher temporal resolution significantly improves semantic separability in zero-shot settings. We further analyze the role of tracking-based human joint information under both full and partial observation scenarios. Quantitative evaluation using a nearest-class prototype strategy demonstrates that high-speed video provides more stable and interpretable semantic representations for fast actions. These findings highlight the importance of temporal resolution in training-free action recognition and suggest that high-speed perception can enhance semantic understanding capabilities. I Introduction Understanding human actions from visual observations is a fundamental problem in human–robot interaction (HRI), particularly for applications requiring timely and proactive responses. In real-world scenarios such as collaborative tasks or sports interactions, robots must interpret human actions from incomplete and rapidly evolving motion cues. This challenge is further amplified for high-speed actions, where critical discriminative information may occur within very short temporal intervals. Recent advances in action recognition have been largely driven by supervised learning methods, including skeleton-based graph convolutional networks and video-based recognition models. While these approaches achieve strong performance, they heavily rely on task-specific training data and often assume standard video frame rates (e.g., 30 FPS). In contrast, zero-shot and training-free approaches, enabled by vision-language models (VLMs) and large language models (LLMs), provide a promising alternative for action understanding without requiring labeled datasets. However, existing works primarily focus on general daily actions and overlook the impact of temporal resolution, especially for fast and fine-grained motions. In this work, we investigate whether high-speed visual input can improve zero-shot semantic discriminability of human actions. Specifically, we select high-speed kendo attack actions as a representative case of rapid and subtle motion patterns, and examine whether their semantic differences can be captured without any task-specific training. We propose a training-free pipeline that leverages a pre-trained video-language model to generate textual descriptions from short video clips, followed by an LLM model for reasoning and measuring pairwise action similarity. Through controlled experiments across different frame rates (120 Hz, 60 Hz, and 30 Hz), we demonstrate that higher temporal resolution significantly enhances semantic separability in zero-shot settings. To further reflect practical human–robot interaction scenarios, we additionally evaluate the proposed framework under partial observation conditions for early action understanding, where only the initial stage of an action is available before completion. Furthermore, we analyze the role of tracking-based human joint information and its interaction with temporal resolution under both full and early-stage observation settings. The main contributions of this work are summarized as follows: • We present a proof-of-concept study on the effect of temporal resolution in zero-shot semantic understanding of high-speed human actions. • We propose a training-free pipeline combining video-language models and LLM-based reasoning for action comparison without task-specific supervision. • We demonstrate that higher frame rates improve pre-trained models’ semantic understanding, and provide insights into when additional cues such as joints tracking are beneficial. II Related Work II-A Human Action Recognition Human action recognition plays an important role in HRI. Conventional approaches commonly rely on skeleton-based representations and deep learning architectures such as Graph Convolutional Networks (GCNs) to capture spatiotemporal motion patterns from human body joints [ 21 , 3 , 4 , 11 , 23 , 7 , 5 , 15 , 8 ] . These methods have demonstrated strong performance on common action recognition benchmarks. However, most existing approaches are training-based and require large amounts of annotated data, which limits their applicability to new, rare or highly specialized actions. II-B Vision–Language Models for Action Understanding Recent advances in VLMs and LLMs have enabled semantic understanding without task-specific training. MotionCLIP [ 18 ] projects human motion into the CLIP [ 14 ] embedding space, enabling semantic alignment between motion and text. Zhang et al. [ 22 ] utilize pre-trained VLMs for contrastive learning, which enhances the alignment between visual features and textual semantics. In addition, LLM-based reasoning methods have recently been adopted for action recognition. LLM-AR [ 13 ] converts human motions into linguistic descriptions and formulates action recognition as a language understanding problem. Similarly, GAP [ 20 ] utilizes LLM to generate semantic descriptions of movements and incorporates these descriptions into action recognition. Building upon these foundations, further research has extended these models to training-free zero-shot action understanding. These approaches leverage semantic reasoning and pre-trained knowledge to recognize actions. FreeZAD [ 6 ] performs action localization and recognition directly, demonstrating that pre-trained VLMs can generalize to unseen actions through semantic matching. Similarly, TEAR [ 1 ] enhances action understanding through action descriptors and predicts actions through similarity matching between textual and visual embeddings. Despite these advances, most VLM-based action understanding studies focus on daily activities captured at standard frame rates. The effect of temporal resolution on semantic understanding of human actions remains largely underexplored, particularly for rapid and fine-grained movements such as martial arts actions. II-C Early Action Recognition under Partial Observation Early action recognition aims to identify human actions before the complete motion sequence becomes available. This capability is essential for real-time robotic systems, where response latency directly affects interaction quality and safety. Existing methods typically perform action prediction from partial observations. [ 16 ] use attention to weight discriminative early motion frames, while [ 9 ] models action-semantic consistent knowledge. However, these approaches generally rely on supervised training and large annotated datasets. In contrast, this work explores whether partial observations of high-speed actions preserve sufficient semantic information for zero-shot understanding. Our experiments further analyze how temporal resolution influences semantic separability under incomplete observation conditions. II-D Kendo Motion Analysis and High-Speed Perception Several studies have investigated human–robot interaction and motion recognition in Kendo. The sword-fighting robot proposed in [ 12 ] estimates the human motion trajectory and generates corresponding defensive actions. Kendo action recognition has also been explored using wearable sensors together with the LSTM method [ 17 ] . Moreover, our previous work [ 2 ] explores markerless high-speed Kendo action tracking and recognition. Different from these training-based or rule-based approaches, this work interprets high-speed Kendo actions under a zero-shot and training-free setting. Instead of relying on handcrafted criteria or wearable sensors, we leverage vision-language models and large language models to perform semantic comparison. III Methodology In this work, we propose a training-free semantic comparison pipeline to investigate whether high-speed kendo actions can be semantically interpreted without task-specific training. Specifically, we explore whether semantic discriminability between different attack