Temporal Realism Evaluation of Generated Videos Using Compressed-Domain Motion Vectors

Mert Onur Cakiroglu, İdil Bilge Altun, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban · 2025

Read Temporal Realism Evaluation of Generated Videos Using Compressed-Domain Motion Vectors by Mert Onur Cakiroglu, İdil Bilge Altun, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13897.txt.

Introduction

Mert Onur Cakiroglu, Idil Bilge Altun, Zhihe Lu, and two co-authors judge AI-generated videos by their motion rather than their looks. Most scoring tools reward still-frame quality and miss unrealistic movement. Their framework reads motion vectors straight from compressed video files, a cheap and consistent record of how each frame shifts.

Comparing motion statistics between real and generated clips with standard distance measures, they rank eight leading generators on the GenVidBench data set. Some models track real motion closely while others stray widely. Their plots expose flaws such as center bias and grid-like flow patterns. They also test ways of fusing motion data into scoring models.

Catalog record

CREATORMert Onur Cakiroglu, İdil Bilge Altun, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temporal Realism Evaluation of Generated Videos Using Compressed-Domain Motion Vectors Mert Onur Cakiroglu, İdil Bilge Altun, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban Abstract Temporal realism remains a central weakness of current generative video models, as most evaluation metrics prioritize spatial appearance and offer limited sensitivity to motion. We introduce a scalable, model-agnostic framework that assesses temporal behavior using motion vectors (MVs) extracted directly from compressed video streams. Codec-generated MVs from standards such as H.264 and HEVC provide lightweight, resolution-consistent descriptors of motion dynamics. We quantify realism by computing Kullback-Leibler, Jensen-Shannon, and Wasserstein divergences between MV statistics of real and generated videos. Experiments on the GenVidBench dataset containing videos from eight state-of-the-art generators reveal systematic discrepancies from real motion: entropy-based divergences rank Pika and SVD as closest to real videos, MV-sum statistics favor VC2 and Text2Video-Zero, and CogVideo shows the largest deviations across both measures. Visualizations of MV fields and class-conditional motion heatmaps further reveal center bias, sparse and piecewise constant flows, and grid-like artifacts that frame-level metrics do not capture. Beyond evaluation, we investigate MV–RGB fusion through channel concatenation, cross-attention, joint embedding, and a motion-aware fusion module. Incorporating MVs improves downstream classification across ResNet, I3D, and TSN backbones, with ResNet-18 and ResNet-34 reaching up to 97.4% accuracy and I3D achieving 99.0% accuracy on real-versus-generated discrimination. These findings demonstrate that compressed-domain MVs provide an effective temporal signal for diagnosing motion defects in generative videos and for strengthening temporal reasoning in discriminative models. The implementation is available at: https://github.com/KurbanIntelligenceLab/Motion-Vector-Learning 1 Introduction Recent breakthroughs in text-to-video and image-to-video generation have enabled the synthesis of high-resolution 13 , visually compelling clips that closely resemble real-world footage. These advances, driven by large-scale diffusion models 11 and transformer architectures 14 trained on large-scale paired video and text corpora, increasingly capture long-range dependencies through spatiotemporal attention and improved conditioning. Despite substantial progress in per frame fidelity, assessing and ensuring temporal realism, defined as motion that is physically plausible, stable, and coherent over time, remains a central challenge for evaluation and downstream use 19 . A growing body of work extends image-generation metrics to video, including Fréchet Video Distance (FVD), Inception Score (IS), and perceptual measures such as LPIPS 22 ; 9 ; 28 . Kernel-based distances, for example kernel video distance, provide additional signal beyond frame statistics 1 . These approaches primarily emphasize spatial appearance and often fail to capture subtle motion artifacts such as jitter, drift, or repetition that strongly affect perceptual quality and task performance 23 . Methods that explicitly model motion, such as optical flow, keypoint trajectories, or motion tracking, are informative but computationally expensive and introduce modeling assumptions that may bias evaluation. Compressed-domain motion vectors (MVs), computed by standard codecs such as H.264 and HEVC during inter-frame prediction, offer a lightweight alternative for characterizing temporal structure. MVs are resolution-consistent, naturally aligned with frame-to-frame motion, and available at negligible cost during decoding. Building on these properties, we show that statistics of MVs provide a complementary signal for analyzing motion realism in generated videos. When combined with RGB colors, they also enhance downstream video classification without requiring dense flow estimation. This perspective connects to the compressed-domain literature in action recognition, for example CoViAR and DMC-Net 27 ; 18 , but is applied here to generative evaluation and practical scalability. We quantify motion realism by comparing distributions of MV-based statistics between real and synthetic videos using divergence measures including Kullback-Leibler (KL), Jensen-Shannon (JS), and Wasserstein distances (WD). We also investigate several fusion strategies that combine MV information with RGB features to enrich spatial representations with temporal cues. Together, these components form an efficient and model-agnostic framework that exposes temporal inconsistencies in generative video models and improves classification robustness in scenarios where synthetic videos contain subtle motion artifacts that are difficult to detect with frame-centric metrics. Figure 1 : Overview of the proposed compressed-domain motion framework. (I) Compressed-domain MVs are obtained from standard video codecs using block matching on macroblocks, for example 16{\times}16 . Frame A illustrates a macroblock, and Frame B shows the search process and the selected best-matching block, which forms the MV between frames. (II) MV fields are spatially aligned with RGB frames to produce consistent motion representations. We consider multi-modal fusion paradigms, including channel concatenation, joint embedding, cross-attention, and motion-aware fusion. In motion-aware fusion, motion masks modulate the contribution of motion features to appearance features, emphasizing informative motion while suppressing noise. Contributions. Our work makes four contributions: (1) We use codec-generated MVs from standard video streams, for example H.264 and HEVC, as an efficient and model-agnostic signal for temporal assessment, providing resolution-consistent motion information at negligible additional cost compared to optical flow; (2) We propose a quantitative framework that measures motion realism using divergence metrics including KL, JS, and WD computed between motion statistics of real and generated videos; (3) We design and compare several fusion strategies that combine MV information with RGB colors, including channel concatenation, cross-attention, joint embedding, and motion-aware fusion; (4) We demonstrate that augmenting RGB with MV channels improves downstream video classification by enriching spatial features with temporal information. 2 Related Work 2.1 Video Generation Models The rapid progress in video synthesis has been driven largely by diffusion-based generative models and transformer architectures trained on large-scale video-text corpora. Approaches like VideoCrafter2 6 , Stable Video Diffusion 2 , and SVD 3 employ spatiotemporal attention mechanisms and latent diffusion priors to produce high-resolution videos with strong perceptual fidelity. Contemporary systems differ widely in their data regimes and architectural choices. For instance, SVD and CogVideo rely on the HD-VG130M dataset 26 containing 1280 \times 720, 30 FPS videos, whereas VideoCrafter2, Pika, and Text2Video-Zero are trained on the VidProm dataset 6 ; 16 ; 11 . These systems also vary in their output characteristics: Pika generates 1088 \times 560 videos at 24 FPS; VideoCrafter2 produces 512 \times 320 videos at 10 FPS; Text2Video-Zero synthesizes 512 \times 512 videos at 4 FPS using training-free motion-aware latent codes; SVD outputs 1024 \times 576 videos at 10 FPS via a three-stage training strategy; and CogVideo generates 480 \times 480 videos at 4 FPS with multi-frame-rate hierarchical generation 2 ; 10 . Despite these architectural innovations and perceptual improvements, evaluating the temporal realism of synthetic motions remains an unsolved challenge. 2.2 Video Quality and Temporal Evaluation Metrics Most classical video evaluation metrics originate from image generation and primarily target frame-level fidelity rather than motion consistency. Metrics such as FVD and IS 22 ; 9 do not adequately capture subtle temporal discontinuities that compromise realism. Although computing Fréchet Distance on spatiotemporal I3D features 1 ; 28 offers improved sensitivity to temporal structure, these methods are computationally expensive and often fail to detect fine-grained inconsistencies crucial for downstream tasks. As diffusion-based models continue to improve in per-frame quality, the lack of reliable temporal realism metrics has become a critical bottleneck for evaluating advances in generative modeling. 2.3 Video Compression and Motion Representations Video compression standards such as AVC/H.264 and HEVC 21 have long used motion modeling to reduce temporal redundancy. These codecs operate through intra-frame prediction and inter-frame motion compensation, leveraging I-, P-, and B-frame structures to efficiently encode temporal changes 20 . Following motion prediction, blocks are transformed using DCT, quantized, and entropy-coded using methods such as

Read the complete text