Mobile-VideoGPT: Fast and Accurate Model for Mobile Video Understanding
Abdelrahman Shaker 1,∗, Muhammad Maaz 1, Chenhui Gou 2, Hamid Rezatofighi 2, Salman Khan 1, Fahad Shahbaz Khan 1,3 · 2025
Read Mobile-VideoGPT: Fast and Accurate Model for Mobile Video Understanding by Abdelrahman Shaker 1,∗, Muhammad Maaz 1, Chenhui Gou 2, Hamid Rezatofighi 2, Salman Khan 1, Fahad Shahbaz Khan 1,3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21782.txt.
Introduction
Abdelrahman Shaker and co-authors present Mobile-VideoGPT in 2025, a small model that understands video. Video models tend to be huge and slow, which blocks use on phones. Their model stays under a billion parameters while keeping accuracy high.
Light encoders, a small language model, and smart frame picking do the work. A scoring step selects key frames, and a projector trims repeated visual tokens. On six standard video tests the small model ran fast and beat larger rivals in the report. Real-time video help on a phone, from search to description, moves closer with designs like this.
Catalog record
| CREATOR | Abdelrahman Shaker 1,∗, Muhammad Maaz 1, Chenhui Gou 2, Hamid Rezatofighi 2, Salman Khan 1, Fahad Shahbaz Khan 1,3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21782
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mobile-VideoGPT: Fast and Accurate Model for Mobile Video Understanding Abdelrahman Shaker 1,∗, Muhammad Maaz 1, Chenhui Gou 2, Hamid Rezatofighi 2, Salman Khan 1, Fahad Shahbaz Khan 1,3 Abstract Video understanding models often struggle with high computational requirements, extensive parameter counts, and slow inference speed, making them inefficient for practical use on edge devices. To tackle these challenges, we propose Mobile-VideoGPT, an efficient multimodal framework designed to operate with fewer than a billion parameters. Unlike traditional video large multimodal models (LMMs), Mobile-VideoGPT consists of lightweight dual visual encoders, efficient projectors, and a small language model (SLM), enabling real-time inference on resource-constrained platforms. To further improve efficiency, we present an Attention-Based Frame-Scoring mechanism to select the key-frames, along with an efficient token projector that prunes redundant visual tokens and preserves essential contextual cues. We evaluate our model across well-established six video understanding benchmarks (e.g., MVBench, EgoSchema, NextQA, and PerceptionTest), and our results demonstrate that Mobile-VideoGPT-0.5B achieves up to 7.3 tokens per second on NVIDIA Jetson Orin edge devices and 45.9 tokens per second on a single NVIDIA RTX A6000. Despite its compact size, it outperforms existing state-of-the-art 0.5B-parameter models by an average of 6 points, while using 40% fewer parameters and delivering more than 2 × \times higher throughput across both edge and desktop platforms. Our code and models are publicly available at: https://github.com/Amshaker/Mobile-VideoGPT . 1 Introduction Figure 1: Left : Performance comparison of Mobile-VideoGPT against state-of-the-art efficient video understanding models across six benchmarks (PercepTest, NextQA, MVBench, EgoSchema, ActivityNet-QA, and MLVU). Mobile-VideoGPT achieves consistently strong accuracy across diverse video reasoning tasks. Right : Efficiency comparison in terms of total parameters, model size, and inference throughput (tokens/second) on Jetson Orin Nano and RTX A6000. Despite using significantly fewer parameters and a smaller memory footprint, Mobile-VideoGPT has higher throughput with better accuracy–efficiency trade-off and strong suitability for edge devices. Multimodal Large Language Models (MLLMs) have emerged as a pivotal area of research and demonstrated exceptional abilities in video understanding tasks, including video captioning, visual question-answering, and visual grounding [ 33 , 39 , 21 ] . Remarkable performance on these tasks can be witnessed in proprietary models such as GPT-4o [ 45 ] and Gemini-1.5-Pro [ 11 ] or open-source alternatives such as Video-LLaVA [ 33 ] and Video-ChatGPT [ 39 ] . Despite the promising performance of these methods across specific benchmarks, deploying these models on edge devices presents a significant challenge. The design of large multimodal architectures often results in a substantial number of parameters and high computational demands, rendering them impractical for resource-constrained environments. For instance, the recently introduced LLaVa-Mini [ 65 ] proposes an image and video MLLM based on a single one‐vision token, yet its total number of parameters is 8.4B, and the model size is more than 20GB, making it impractical for real-time performance and edge deployment. Hence, there is a need for efficient video understanding models and training paradigms that are capable of retaining strong performance without exceeding the resource limitations. Over the past decade, techniques such as knowledge distillation [ 16 , 51 ] , model pruning [ 14 , 15 ] , efficient attention [ 52 , 38 ] , and parameter-efficient fine-tuning [ 17 ] have partially addressed the challenge of reducing model complexity under resource constraints. However, these strategies come with notable performance trade-offs or require specialized hardware accelerators. Moreover, while they may effectively reduce the size of individual components (e.g., visual or text encoders), ensuring efficient projection and integration across modalities for video understanding and with advanced capabilities remains a challenge. To this end, we introduce Mobile-VideoGPT, an efficient video understanding language model that balances between robust performance and high throughput on edge devices (i.e, Jetson Orin Nano [ 42 ] ). We demonstrate the efficiency and the effectiveness of Mobile-VideoGPT across multiple video conversation benchmarks, including ActivityNet-QA, EgoSchema, MLVU, MVBench, NextQA, and PerceptionTest, where it not only is faster than the previous SoTA, but also outperforms them (See Fig. 1 ). Our contributions in this work are as follows: 1. A lightweight multimodal model designed specifically for real-time video understanding with a focus on edge device deployment. 2. We propose a novel frame scoring and selection strategy that optimally selects the representative video key-frames, along with efficient token projection, to enhance the model’s ability to maintain visual fidelity while reducing computational overhead for efficient processing. 3. Our architecture is carefully optimized for edge deployment, featuring a compact model size of just 1GB in FP16 precision and requiring only 3GB of VRAM. Owing to its efficient design, it achieves a high throughput of 7.3 tokens per second on Jetson Orin Nano [ 42 ] . 4. Mobile-VideoGPT showcases superior performance compared to competitive models across six video understanding benchmarks, achieving an average improvement of 6 points over LLaVA-OneVision-0.5B [ 21 ] . Our model is also 2 \times faster, while being less than 3 \times its size. 2 Related Work Small Language Models (SLMs) : In the past year, the rise of large language models (LLMs) has driven significant progress in natural language processing, including proprietary models like GPT-4 [ 43 ] , and open-source models such as LLaMA [ 55 ] and DeepSeek [ 34 ] . Although these models have sparked increased interest in enhancing the performance of LLMs, the large size of these models often leads to substantial memory consumption and slow processing speeds. Moreover, the energy demands associated with training and operating these models raise important sustainability issues, which are becoming increasingly urgent as LLMs expand in scale and complexity. To address this, there has been a growing trend in the development of small and medium-sized LLMs, such as Qwen2-0.5B [ 59 ] and Llama3.2-1B [ 1 ] , specifically designed for resource-constrained environments. Given the limited computational capabilities of edge devices such as mobile phones, autonomous vehicles, and embodied AI systems, several lightweight LLMs [ 30 , 64 , 37 , 7 ] have gained significant attention. TinyLLaMA [ 64 ] conducts great open-source work on their 1.1B base and chat models with the same architecture and tokenizer as LLaMa2 [ 54 ] . MobileLLaMA [ 7 ] scales down the architecture of LLaMA and releases 1B and 3B models trained from scratch with public datasets. Recently, MobileLLM [ 37 ] optimizes sub-billion parameter models by proposing an immediate block-wise weight-sharing approach. These compact models are optimized to deliver efficient performance while addressing the unique constraints of resource-limited environments. Another vital line of research involves applying model compression techniques for LLMs, such as pruning [ 9 ] , quantization [ 57 , 27 ] , and low-rank decomposition [ 60 ] . These techniques reduce the computational demands of LLMs and enhance their practicality for deployment on resource-constrained devices. Vision Language Models : The advancement of visual language models (VLMs) has been significantly shaped by the introduction of CLIP [ 48 ] , which utilizes a contrastive loss to align vision and language embeddings within a unified representation space. Recent progress in LLMs has further enhanced the development of VLMs, as demonstrated in Kosmos-2 [ 47 ] , GLaMM [ 49 ] , BLIP-2 [ 22 ] , MiniGPT-4 [ 68 ] , LLaVA [ 36