Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos
Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram · 2025
Read Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos by Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01803.txt.
Introduction
Xavier Thomas and co-authors assess human motion in synthesized videos with action tell-tales. Video generators improve fast, but judging motion correctness is hard. Vision encoders and multimodal models lean on appearance. They miss time dynamics and flag few anatomical errors.
Their Generative Action Tell-Tales target motion itself. Checks focus on how acts unfold over time. Motion-aware scores help rank generators for film and training data. They also expose which acts, like jumps or lifts, current models still fake poorly. Tell-tales focus on joints and limbs over time, the exact cues coaches use when they judge whether a filmed move is real.
Catalog record
| CREATOR | Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01803
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram Abstract Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-vision encoders and Multimodal Large Language Models (MLLMs) are strongly appearance-biased, lack temporal understanding, and thus struggle to discern intricate motion dynamics and anatomical implausibilities in generated videos. We tackle this gap by introducing a novel evaluation metric derived from a learned latent space of real-world human actions. Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features. We posit that this combined feature space provides a robust representation of action plausibility. Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution. For rigorous validation, we develop a new multi-faceted benchmark specifically designed to probe temporally challenging aspects of human action fidelity. Through extensive experiments, we show that our metric achieves substantial improvement of more than 𝟔𝟖 % \mathbf{68\%} compared to existing state-of-the-art methods on our benchmark, performs competitively on established external benchmarks, and has a stronger correlation with human perception. Our in-depth analysis reveals critical limitations in current video generative models and establishes a new standard for advanced research in video generation. Code is available at xthomasbu.github.io/video-gen-evals Figure 1 : What are the telltale signs of a generative action video ? We answer this by learning a robust manifold based on appearance and anatomical coherence exhibited by humans performing actions across several real-world videos. This manifold serves as anchors against which we project the features of a generated video in question and assess its realism. 1 Introduction How do humans learn the right way to perform an action, like walking or making a toast ? Since infancy, we learn by implicitly observing and explicitly being taught by others, grasping the flow of events and the physical laws governing human motions and actions 2 . This intuitive understanding allows humans to effortlessly recognize motion inconsistencies even in today’s highly photorealistic generated videos 43 ; 51 ; 59 . This raises a critical question: Can we formalize our intuitive perception and design a framework to systematically evaluate the accuracy of human actions in generated videos? This is a challenging task, as it requires solving a twofold challenge. First, the baseline problem of recognizing action correctness is ill-posed, even in real videos. Actions can be atomic 21 (e.g., “walking”) or procedural (e.g., “making a toast”) 8 , making automatic recognition inherently challenging. Second, for generated videos, this problem is magnified as we must move beyond detecting the mere presence of an action to also evaluating the temporal coherence of the body movements over time. Current metrics for judging generated videos, such as pixel-level similarity 28 ; 61 , perceptual quality 68 ; 55 , or text-to-video prompt alignment 25 ; 38 , as well as recent approaches that use MLLMs as judges 24 ; 6 , do not accurately capture the complex motion physics of human actions 52 ; 15 . We posit that this is because the underlying representations fundamentally lack awareness of subtle anatomical violations or temporal incoherence. A core contribution of our work is to bridge this gap by building a robust assessment tool that moves beyond superficial statistics and bakes in the critical awareness of physical and anatomical-consistency of human motion. Our key idea is to learn a latent manifold of natural human motion. This manifold is built from semantic features that measure the consistency of human anatomy, motion physics, and visual appearance in a video. As illustrated in Fig. 1 , we learn this manifold from a large volume of diverse videos capturing a wide range of body geometries and performance styles (e.g., a tall woman walking vs. an older male with a walking stick), thereby encapsulating the crucial cues of natural action. To evaluate a new video, we project its embeddings into this manifold and systematically measure the deviations and discern telltale signs of poor action quality, correctness, and coherence. While several benchmarks exist 71 ; 38 , they fail to adequately probe for the fine-grained temporal correctness and coherence of human actions. We identify this as a critical limitation and create an open-source benchmark we call the “ T elltale A ction G eneration Bench” ( TAG-Bench ). We generate hundreds of videos from state-of-the-art open- and closed-source models and conduct a large-scale subjective study. We evaluate the videos on two key criteria: (a) whether the generated video captures the intended action, and (b) the temporal smoothness and anatomical plausibility of the perceived action. Our extensive experiments on TAG-Bench and other benchmarks 71 yield several key findings. First, we highlight a critical gap in current evaluation: we show that all state-of-the-art models and MLLMs struggle to correctly evaluate action correctness and temporal coherence. This demonstrates the sheer difficulty of the task. We further find that certain actions are challenging for all generative models, revealing broader limitations in current video synthesis capabilities. Second, we demonstrate that, unlike existing methods, our manifold’s scores strongly align with human opinion, across both image-to-video (I2V) 69 and text-to-video (T2V) 10 generation settings, and across multiple models. In summary, our contributions are: • We design a learned latent space that encodes human body geometry, motion smoothness, and temporal coherence to evaluate action quality in generated videos. • We design TAG-Bench , a new benchmark with human ratings focused on the correctness and temporal coherence of human actions in generated videos. • We propose two metrics that align closely with human perception. They measure the consistency and the temporal plausibility of actions. • We provide an in-depth analysis of the proposed latent action manifold, and validate the design choices that led to its robust performance. 2 Related Work Video generation models. Video generation 51 ; 27 ; 32 ; 59 ; 44 has rapidly advanced, producing increasingly photorealistic and temporally coherent content 64 ; 26 . Beyond visual fidelity, recent work 62 frames these models as emerging world models capable of capturing complex real-world dynamics without explicit supervision. Despite such progress, current systems still often generate human motion that is kinematically implausible or semantically incorrect 52 ; 15 . This highlights the need for metrics that specifically diagnose and quantify these failures. Distribution and reference based metrics. Metrics such as FVD 55 measure statistical similarity between real and generated videos in pretrained feature spaces, capturing coarse spatiotemporal alignment but missing fine-grained semantic or physical accuracy. Frame-level measures including PSNR 28 , SSIM 61 , and LPIPS 68 assess visual similarity to reference frames but ignore temporal coherence critical to video perception. Recent efforts like Physics-IQ 43 measure spatiotemporal realism using real-world references, but do not focus on capturing human-body distortions or action-level correctness. Reference-free video metrics. In many scenarios, ground-truth videos are unavailable, motivating the need for reference-free evaluation. CLIPScore 25 measures frame-text similarity scores but only captures single-frame semantics, lacking motion or physical plausibility. More recent methods employ MLLMs for richer video understanding, via zero-shot prompting or fine-tuning on human ratings.