ExAct : A Video-Language Benchmark for Expert Action Analysis
Han Yi, Yulu Pan, Feihong He, Xinyu Liu, Benjamin Zhang, Gedas Bertasius · 2025
Read ExAct : A Video-Language Benchmark for Expert Action Analysis by Han Yi, Yulu Pan, Feihong He, Xinyu Liu, Benjamin Zhang, Gedas Bertasius (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06277.txt.
Introduction
Han Yi, Yulu Pan, Feihong He, Xinyu Liu, and Benjamin Zhang lead a team on this paper from two thousand twenty-five, whose text adds Oluwatumininu Oguntola. It presents ExAct, a video and language benchmark for expert grasp of skilled bodily acts.
The set holds thousands of expert-made question pairs over eleven acts in six fields, from sports and cooking to dance and bike repair. Each item gives five close choices, so a fine read is a must. The best model named in the paper scores far below trained human experts. The gap shows how far vision models sit from a pro's eye for craft. Data and code are public.
Catalog record
| CREATOR | Han Yi, Yulu Pan, Feihong He, Xinyu Liu, Benjamin Zhang, Gedas Bertasius |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06277
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ExAct : A Video-Language Benchmark for Expert Action Analysis Han Yi, Yulu Pan, Feihong He, Xinyu Liu, Benjamin Zhang, Gedas Bertasius Abstract We present ExAct , a new video-language benchmark for expert-level understanding of skilled physical human activities. Our new benchmark contains 3,521 expert-curated video question-answer pairs spanning 11 physical activities in 6 domains: Sports, Bike Repair, Cooking, Health, Music, and Dance. ExAct requires the correct answer to be selected from five carefully designed candidate options, thus necessitating a nuanced, fine-grained, expert-level understanding of physical human skills. Evaluating the recent state-of-the-art VLMs on ExAct reveals a substantial performance gap relative to human expert performance. Specifically, the best-performing GPT-4o model achieves only 44.70% accuracy, well below the 82.02% attained by trained human specialists/experts. We believe that ExAct will be beneficial for developing and evaluating VLMs capable of precise understanding of human skills in various physical and procedural domains. Dataset and code are available at https://texaser.github.io/exact_project_page/ . Figure 1 : An illustration of several multiple-choice question samples from our expert action analysis benchmark, ExAct . Here, we visualize samples from three domains of skilled activities (i.e., basketball, bike repair, and piano). The correct answers have a green checkmark next to them and were obtained using domain-expert/coach annotations. The phrases in green (correct) and red (incorrect) emphasize the subtle yet critical differences between ground-truth expert descriptions and incorrect candidate answers. 1 Introduction Dataset Expert-level Knowledge Free-form Language Annotations MCQ Evaluation Coarse Action Recognition Datasets Kinetics-700 [ 8 ] ✗ ✗ ✗ HowTo100M [ 33 ] ✗ ✗ ✗ UCF101 [ 45 ] ✗ ✗ ✗ HMDB [ 21 ] ✗ ✗ ✗ Moments in Time [ 34 ] ✗ ✗ ✗ Hollywood [ 43 ] ✗ ✓ ✗ ActivityNet-QA [ 51 ] ✗ ✗ ✓ Fine-grained Action Recognition Datasets Something-SomethingV2 [ 15 ] ✗ ✗ ✗ FineGym [ 42 ] ✗ ✗ ✗ Multisports [ 25 ] ✗ ✗ ✗ TemporalBench [ 6 ] ✗ ✓ ✓ Video-Based Skill Assessment Datasets JIGSAWS [ 1 ] ✓ ✗ ✗ Best [ 12 ] ✓ ✗ ✗ FineDiving [ 50 ] ✓ ✗ ✗ FP-Basket [ 4 ] ✓ ✗ ✗ BASKET [ 36 ] ✓ ✗ ✗ Aifit [ 13 ] ✓ ✓ ✗ Skilled Activity Video-Language Datasets VidDiffBench [ 5 ] ✓ ✓ ✗ EgoExo-Fitness [ 26 ] ✓ ✓ ✗ EgoExolearn [ 19 ] ✓ ✓ ✗ Ego-Exo4D [ 16 ] ✓ ✓ ✗ ExAct (Ours) ✓ ✓ ✓ Table 1 : Compared to previous action recognition and skill assessment datasets, our proposed ExAct benchmark uniquely combines expert-level, free-form language annotations and a multiple-choice question (MCQ) evaluation format, making it an excellent resource for evaluating modern video-language models at expert-level understanding of skilled human activities. Today, learning and perfecting a new physical skill requires a significant amount of time, practice, and often guidance from an expert coach/professional in that domain. Unfortunately, personalized coaching remains inaccessible to the majority due to prohibitively expensive costs and a lack of expert availability. Recent advances in AI have sparked interest in developing virtual AI assistants/coaches, particularly in the text domain [ 35 , 2 , 11 ] . However, text-based large language models (e.g., ChatGPT) are insufficient for learning new physical skills as standard LLMs typically lack a nuanced understanding of physical human activities/skills. In contrast, rapidly improving vision-language models (VLMs) could serve as valuable tools for various physical skill learning applications. In particular, by recording a video of a skill demonstration and feeding it to a VLM, people could receive detailed, actionable feedback similar to that of expert human coaches. Although recent progress has enabled modern VLMs to achieve impressive general image/video recognition capabilities [ 20 , 46 , 29 , 3 , 27 , 9 ] , existing studies reveal that such VLMs still struggle to understand fine-grained human activities [ 6 , 36 ] , particularly activities that require expert-level knowledge [ 5 , 19 , 26 ] . This is primarily due to the inadequate underlying visual representations that 1) do not capture expert-level knowledge needed to generate feedback for physical skill learning, and 2) the inability to recognize subtle details in skilled human actions. In addition to the limitations of existing VLMs, there is a notable lack of evaluation benchmarks tailored for expert-level understanding of skilled human activities. As shown in Table 1 , most existing datasets focus on coarse activity recognition [ 8 , 33 , 45 , 21 , 34 , 43 , 51 ] , which typically only require scene-level recognition rather than fine-grained understanding. While several fine-grained video recognition datasets exist [ 15 , 42 , 25 , 6 ] , the majority of them are aimed at generic ( e.g. , putting something into something) rather than expert-level action understanding of skilled human activities ( e.g. , keeping balance during a dance spin, playing the correct rhythm in a piano piece). Beyond action recognition, a number of video-based skill assessment benchmarks have been recently developed [ 1 , 12 , 50 , 4 , 36 , 13 ] . Most of such skill assessment datasets focus on predicting scalar/categorical performance scores to quantify execution quality. Additionally, these datasets often lack open-ended language annotations, which can provide a rich and intuitive medium to convey skill-specific feedback [ 5 ] . Finally, a few recent skilled activity video-language datasets [ 5 , 26 , 19 , 16 ] use experts to obtain free-form language descriptions akin to verbal coach feedback but do not provide rigorous evaluation benchmarks or tasks, making it difficult to assess how well modern video models can understand nuanced physical human skills. Figure 2 : Left: Our proposed ExAct benchmark contains 11 skilled activity types spanning 6 broader physical domains: Sports, Music, Dance, Health, Cooking, and Bike Repair. Top Right: Distribution of video lengths across the dataset, showing that most clips fall within the 0–10 second range. Bottom Right: Sample distribution per activity, categorized by the expert feedback type: Good Execution (GE) and Tips for Improvement (TIPS). To address these issues, we introduce ExAct , a video-language benchmark consisting of 3,521 video–question–answer (VQA) pairs designed to evaluate expert-level understanding of skilled physical human actions (sample questions shown in Figure 1 ). ExAct covers 11 activities across 6 diverse physical domains: Sports (Basketball, Soccer, Bouldering), Bike Repair, Cooking, Health (COVID - 19 Safety, CPR), Music (Guitar, Piano, Violin), and Dance, as shown in Figure 2 . To construct ExAct , we build on the fine-grained expert commentaries derived from the Ego-Exo4D dataset [ 16 ] . The original Ego-Exo4D expert commentaries have several crucial limitations: 1) the original expert commentaries are unstructured and lengthy, 2) they contain automatic speech recognition (ASR) errors, and 3) include redundant or irrelevant content that is not directly tied to the observed actions. Furthermore, evaluating the quality of open-ended captions/descriptions in the original form of Ego-Exo4D expert commentaries is inherently difficult using standard language metrics such as CIDEr [ 47 ] , BLEU [ 37 ] , or ROUGE [ 28 ] , which do not accurately reflect the correctness or relevance of the instructional feedback in the context of skilled physical activities. To address these challenges, we first apply a