Efficient Motion-Aware Video MLLM

Zijia Zhao, Yuqi Huo, Tongtian Yue, Longteng Guo, Haoyu Lu, Bingning Wang, Weipeng Chen, Jing Liu · 2025

Read Efficient Motion-Aware Video MLLM by Zijia Zhao, Yuqi Huo, Tongtian Yue, Longteng Guo, Haoyu Lu, Bingning Wang, Weipeng Chen, Jing Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13016.txt.

Introduction

Zijia Zhao and co-authors build a video-language model that reads motion straight from compressed video, in a 2025 paper. Most rival models sample frames evenly and treat each as a still, wasting effort and missing movement. Their model, EMA, encodes each group of pictures as one unit, fusing appearance and motion vectors into compact tokens. A few rich color frames mix with many sparse motion maps.

The team also built MotionBench, a test spanning straight, curved, spinning, and contact motion. EMA topped that bench and the standard video question tests while costing less to run. It also held up on long videos. The authors say compressed structure is a feature, not a nuisance, for video AI.

Catalog record

CREATORZijia Zhao, Yuqi Huo, Tongtian Yue, Longteng Guo, Haoyu Lu, Bingning Wang, Weipeng Chen, Jing Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Motion-Aware Video MLLM Zijia Zhao, Yuqi Huo, Tongtian Yue, Longteng Guo, Haoyu Lu, Bingning Wang, Weipeng Chen, Jing Liu Abstract Most current video MLLMs rely on uniform frame sampling and image-level encoders, resulting in inefficient data processing and limited motion awareness. To address these challenges, we introduce EMA , an E fficient M otion- A ware video MLLM that utilizes compressed video structures as inputs. We propose a motion-aware GOP (Group of Pictures) encoder that fuses spatial and motion information within a GOP unit in the compressed video stream, generating compact, informative visual tokens. By integrating fewer but denser RGB frames with more but sparser motion vectors in this native slow-fast input architecture, our approach reduces redundancy and enhances motion representation. Additionally, we introduce MotionBench, a benchmark for evaluating motion understanding across four motion types: linear, curved, rotational, and contact-based. Experimental results show that EMA achieves state-of-the-art performance on both MotionBench and popular video question answering benchmarks, while reducing inference costs. Moreover, EMA demonstrates strong scalability, as evidenced by its competitive performance on long video understanding benchmarks. 1 Introduction Understanding video content is essential for comprehensive world modeling. Recent advancements in multimodal large language models (MLLMs) for video understanding leverage uniform frame sampling [ 43 , 23 , 12 , 22 , 28 , 39 , 45 , 18 , 31 ] and apply vision encoders [ 42 , 29 ] , which are originally designed for image processing, to analyze each video frame. This approach concatenates frame features temporally before inputting them into the LLM for complex analysis. Here, the vision encoder captures spatial semantics per frame, while the LLM primarily handles temporal comprehension. However, does this framework genuinely achieve comprehensive video understanding? Figure 1 : Comparison of sampling from decoded frames versus the GOPs (Group Of Pictures) from compressed video stream. Compressed video encoding generates tokens at only 1/T the length of sampled frames for the same clip, while capturing motion information more directly. It also shows greater efficiency on Video-QA (average of MSVD-QA and MSRVTT-QA [ 35 ] ) and MotionBench within our EMA framework, achieving higher accuracy with less inference time ( red arrow shows inference speed up). The core difference between images and videos is that, unlike static images, videos contain motion information, which previous methods struggle to handle effectively. As illustrated in Fig. 1 , consider a short segment from a video showing a person’s hand holding a cup. When using uniform frame sampling, challenges arise: sampling a single frame ( e.g . , frame i ) may make it difficult, even for humans, to discern the cup’s movement direction. To better capture motion, denser sampling is needed (( e.g . , frame i\sim i+T )), producing a longer sequence of visual tokens ( e.g . , T\times N ), which increases inference costs. Some methods [ 22 , 37 ] improve efficiency in long video analysis by reducing the number of visual tokens per frame, often at the cost of frame-level semantic detail. Other approaches [ 38 , 47 , 5 ] exploit the redundancy between neighboring frames, employing a slow-fast encoding strategy to process large numbers of frames while compressing tokens through dynamic pooling. These methods mainly address the redundancy in individual frame features due to the similarity between adjacent frames in the video sequence. However, such redundancy can also be reduced prior to visual encoding. We approach video encoding from a different perspective of video stream inputs. In conventional electronic video codecs ( e.g . , MPEG-4, H.264), videos are stored with high compression ratios, significantly reducing redundancy. The compressed video stream holds enough information to fully reconstruct the video into frames, showing it contains what’s needed to understand the video. Compression is achieved by segmenting video data into a limited set of RGB frames (I-frames) and a larger set of sparse motion vectors and residuals (P/B-frames). Most frames in a video sequence are reconstructed by referencing an I-frame and applying corresponding motion vectors and residuals. Using compressed videos [ 17 , 4 , 9 , 11 , 10 , 30 , 13 ] as input (with dense RGB values in I-frames and sparse motion vectors in P/B-frames) naturally forms a slow-fast input structure. This architecture minimizes redundancy compared to a fully decoded frame stream, lessening the need to compress redundant frames. Notably, the motion vectors inherently track the trajectories of macroblocks within each frame, making this compression-domain structure highly effective for capturing motion dynamics. Building on this concept, we introduce an E fficient M otion- A ware video understanding MLLM EMA , which separately encodes key frames and motion frames from compressed video streams. Inspired by modern codecs’ frame decoding, we propose a GOP encoder module to encode and integrate RGB and motion data, which segments video into GOP units and fuses spatial and motion semantics into the same number of visual tokens as a single frame while containing more information inside a video clip. The GOP feature sequence is then fed into a large language model for video understanding via a two-stage training. To evaluate motion understanding, we created MotionBench, a benchmark assessing four common types of motion: linear, curved, rotational, and contact-based. Our model achieved superior performance on both MotionBench and public benchmarks, including MSVD-QA [ 35 ] , MSRVTT-QA [ 35 ] , and ActivityNet-QA [ 41 ] , while demonstrating reduced inference costs. Furthermore, when evaluated on long video understanding benchmarks, such as VideoMME [ 6 ] , the model maintained competitive performance. These experimental results indicate that our approach not only lowers the inference cost of Video MLLMs but also enhances motion capture and overall video understanding. Our contribution can be concluded as: • We introduce EMA , an efficient motion-aware video model that leverages compressed video structures as slow-fast input to enhance motion capture and minimize input redundancy. • Recognizing the lack of benchmarks dedicated to motion understanding, we develop MotionBench, a custom evaluation benchmark that assesses video motion understanding. • Our model achieves competitive results on both MotionBench and public datasets with reduced inference costs. Additionally, it performs well on long video understanding benchmarks, demonstrating good context scalability. 2 Related Works 2.1 Video MLLMs Recent developments in multimodal large language models have extended their applications to video data [ 39 , 43 , 31 ] , where uniform frame sampling and frame-by-frame encoding are commonly employed. Early models, such as VideoChat [ 19 ] , VideoChatGPT [ 28 ] , and Valley [ 27 ] , rely on mean pooling to aggregate frame-level features before feeding them to LLMs, which limits temporal comprehension. Furthermore, some advanced models have introduced specific enhancements to capture dynamic content; for instance, Chat-UniVi [ 12 ] utilizes dynamic token clustering, and BT-Adapter [ 25 ] incorporates lightweight adapters to improve image LLMs for video data. VideoChat2 [ 18 ] further replaces CLIP [ 29 ] with a dedicated video encoder [ 20 ] , achieving better temporal modeling but at the cost of complex, multi-stage training. Video-LaVIT [ 13 ] decomposes video content into keyframes and motion discrete vectors, capturing temporal dynamics through a tokenization strategy. By contrast, Our approach fundamentally departs from prior methods by directly

Read the complete text