Video Action Differencing
James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy · 2025
Read Video Action Differencing by James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07860.txt.
Introduction
James Burgess and co-authors define a new video task in a 2025 paper: spotting how two people perform the same action differently. A coach comparing two golf swings needs those fine differences named and located in time. The team built a benchmark of five hundred forty nine video pairs with thousands of human notes on differences and their timestamps.
Top multimodal models struggled badly on the benchmark. The authors trace the failures to two skills: finding the right sub actions across two videos, and comparing frames finely. Their fix is a three stage agent workflow that proposes differences, locates keyframes, then compares frames with specialist models. Data and code are public.
Catalog record
| CREATOR | James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Action Differencing James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy Abstract How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has many applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-grained action differences and 2,075 localization timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: localizing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, we propose the VidDiff method, an agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To encourage future research in this new task, we release the benchmark 1 1 1 Benchmark: https://huggingface.co/datasets/jmhb/VidDiffBench and code 2 2 2 Project page: http://jmhb0.github.io/viddiff . 1 Introduction The ability to compare two videos of the same action and discern their detailed differences plays a critical role in a wide variety of applications. For instance, in fitness coaching, a novice learning to perform a barbell squat typically watches instructional videos and then compares their actions in a recorded video to identify discrepancies between their movements and those of an expert. In medical training, junior surgeons compare videos of themselves performing surgical procedures with reference videos from experts to identify errors and improve surgical skills. There are two critical obstacles. First is precise localization of sub-actions : finding differences requires finding the sub-action frames where the differences might occur, and aligning those frames between the two videos. Second is fine-grained understanding : the ability to perceive subtle visual differences in motions. Current research on video difference understanding largely emphasizes feature visualization ( Balakrishnan et al. 2015 ) or coarse-grained comparisons between different actions or interacting objects ( Nagarajan & Torresani 2024 ) . However, many real-world applications demand fine-grained comparisons between videos of the same action, a challenge that has received little attention. Figure 1: The Video Action Differencing task and benchmark (VidDiffBench). Given a pair of videos and an action, the task is to generate a list of differences as natural language descriptions. Our VidDiffBench consists of annotated differences across diverse domains, where the differences are relevant to human skill learning. The first row emphasizes the first key challenge: localization of sub-actions between segments of the video for comparison. The second row highlights the second key challenge: fine-grained image understanding of actions in order to perform comparison. We introduce a new task, Video Action Differencing (VidDiff). Given two videos of the same action, (v_{A},v_{B}) , along with a description of the action, the task is to generate two sets of statements: one that is more true for v_{A} and another for v_{B} . For example, in a video pair featuring an expert and a novice performing a barbell squat, key differences might include “knees caving in more in video A” and “the squat is deeper in video B” ( Figure 1 ). Since generating the initial difference candidates relies heavily on language capabilities, we also introduce a simpler ‘closed’ setting that focuses on video analysis. In this setting, the target difference strings are provided, and the task is to predict whether each applies more to video A or B. To facilitate research in this new direction, we present VidDiffBench, a comprehensive benchmark designed for video action differencing. VidDiffBench contains 549 video pairs drawn from domains that require expert feedback, such as fitness, sports, music, and surgery. The videos are annotated with 4,469 fine-grained differences ( \sim 8 per video pair), along with 2,075 timestamp annotations that identify where these differences occur. To ensure the annotated differences are relevant to skill learning, we create a taxonomy of action differences that leverages domain expertise. This makes VidDiffBench the first large-scale dataset dedicated to video action differencing. In addition to introducing a new task and benchmark, we propose the VidDiff Method, an agentic workflow ( Anthropic 2025 ) for solving VidDiff in zero-shot. The method incorporates large language models (LLMs) to propose differences, localizes relevant frames using contrastive language-image models (CLIP), and compares frames for differences using vision-language models (VLMs). The key idea is that localizing specific video segments where differences occur enables more effective visual comparison with VLMs. We further benchmark both open-source (Qwen2-VL, LLaVA-Video) and proprietary (GPT-4o, Gemini-1.5 pro, Claude 3.5 Sonnet) large multimodal models (LMMs) on VidDiffBench. Our results demonstrate that VidDiff is a very challenging task for zero shot models, while the structured approach in the VidDiff Method enhances video comparison. 2 Related Work Skilled Action Understanding in Videos Video comparison has many potential applications, and our benchmark focuses on the specific goal of natural language feedback in skill learning. Most of the video action comparison papers from this section’s first paragraph are systems for skill feedback, showing that skill feedback is well-motivated. Many works give feedback by classifying coarse motion errors, or by visualizing motions, with applications in yoga ( Zhao et al. 2022 ; Thoutam et al. 2022 ; Chen et al. 2018 ; Dittakavi et al. 2022 ; Chen & Yang 2020 ; Xie et al. 2019 ) , physical therapy ( Velloso et al. 2013 ) , weightlifting ( Parmar et al. 2022 ; Ogata et al. 2019 ) , and general fitness ( Fieraru et al. 2021 ; Ashwin et al. 2023 ) . The feedback tends to be coarse-grained. In contrast, our task focuses on open natural language feedback, and identifying fine-grained feedback. Recently, the Ego-Exo4D dataset ( Grauman et al. 2023 ) provides videos with expert commentary on skilled actions, which is promising for developing instructional feedback systems. This, along with existing works that give language feedback ( Li et al. 2024b ; Fieraru et al. 2021 ; Parmar et al. 2022 ; Velloso et al. 2013 ) , support our claim that language is a good medium for providing skill feedback to humans. Zooming out from skills feedback, skilled action understanding – which includes foundational capabilities for feedback systems – has attracted enormous interest. For example, in sports, music, dance, and surgery, prior works have tackled action recognition ( Verma et al. 2020 ; Shahroudy et al. 2016 ; Soomro et al. 2012 ; Zhang et al. 2013 ; Wang & Zemel 2016 ; Chung et al. 2021 ) ; spatial and temporal action localization / segmentation ( Shao et al. 2020 ; Liu et al. 2022 ; Li et al. 2021b ; Zhang et al. 2023b ; Ibrahim et al. 2016 ; Garrow et al. 2021 ; Li et al. 2021b ; Aklilu et al. 2024 ) ; human pose and motion estimation / reconstruction ( Cai et al. 2022 ; Tang et al. 2023b ; Wang et al. 2023 ; Andriluka et al. 2014 ; Li et al. 2021a ; Fieraru et al. 2021