QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design

2025

Read QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16175.txt.

Introduction

Benjamin Schneider, Dongfu Jiang, Chao Du, Tianyu Pang, and Wenhu Chen make hour-long video digestible for video language models in real time. Two bottlenecks block long-video understanding. Decoding the raw stream into frames can eat a minute before any thinking starts. Prefilling millions of tokens then floods memory and stalls inference.

QuickVideo attacks both with a system-algorithm co-design. Its decoder splits video at keyframes and decodes intervals in parallel on the processor. Its prefiller prunes the key-value cache, fitting more frames in less graphics memory. An overlapping schedule runs decoding and inference concurrently instead of in sequence. Together the pieces cut about a minute from long-video inference. Experiments across durations and sampling rates show the gains generalize. Surveillance, lectures, meetings, and sports are the named targets.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design Thanks: The first two authors have equal contribution. Abstract Long video understanding has emerged as a crucial capability in real-world applications such as meeting summarization, video surveillance, educational lecture analysis, and content moderation. However, it remains computationally prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential video decoding , the process of converting the raw bit stream to RGB frames can take up to a minute for hour-long video inputs, and 2) costly prefilling of up to several million tokens for LLM inference, resulting in high latency and memory use. To address these challenges, we propose QuickVideo , a system-algorithm co-design that substantially accelerates long video understanding to support real-time downstream applications. It comprises three key innovations: QuickCodec , a parallelized CPU-based video decoder that achieves 2–3 × \times speedup by splitting videos into keyframe-aligned intervals processed concurrently. QuickPrefill , a memory-efficient prefilling method using KV-cache pruning to support more frames with less GPU memory; and an overlapping scheme that overlaps CPU video decoding with GPU inference. Together, these components reduce the time required to process a long video input by a minute, enabling fast, efficient video understanding even on limited hardware. Experiments show that QuickVideo generalizes across durations and sampling rates, making long video processing feasible in practice. https://github.com/TIGER-AI-Lab/QuickVideo 1 Introduction Video data has become the dominant modality for conveying information online. As of 2023, video data accounts for two thirds of all data transmitted over the Internet [ 30 ] . Much of this data is “long video” ranging from minutes to hours in duration, from online conferencing, gaming, social networking, and movie streaming. This torrent of online video data demands efficient and automated understanding for problems such as content moderation [ 2 ] , real-time surveillance [ 43 ] , and accessibility [ 22 ] . Video Large Language Models (VideoLLMs) [ 4 , 48 , 7 ] have emerged as powerful tools to support these downstream tasks. By natively processing entire video inputs, VideoLLMs exhibit phenomenal potential to understand and reason about video content, offering a practical solution for managing and extracting information from the exponentially growing flood of video data across the Internet [ 47 ] . However, using VideoLLMs for long video understanding suffers from several efficiency challenges. First, the entire video must be decoded from raw bitstreams into RGB frames before the model can begin processing. Current frameworks require up to a minute to decode the frames from an hour-long video input, introducing high latency before any context prefill can start. Second, the prefilling step itself is both computationally and memory intensive [ 37 ] . Each frame—representing an instantaneous moment—can consume hundreds of tokens in the model context [ 49 , 8 ] . As a result, even a modest frame rate (e.g., 2 FPS) for an hour-long video can lead to millions of tokens, far exceeding the memory budget of standard GPUs. Qwen2.5-VL [ 4 ] introduced several architecture modification to accelerate video processing. However, using Qwen2.5-VL-7B, prefilling an hour-long HD video sampled at its native 2 fps still requires more than the 80 GB of memory offered by an A100/H100 SXM4 GPU. Even after reducing the frame sampling rate by 4 \times , prefilling still takes over 25 seconds on datacenter-grade hardware. These inefficiencies result in a frustrating user experience, characterized by long delays and prohibitively high hardware requirements. Users with limited computational resources are effectively excluded from accessing the long video understanding capabilities of VideoLLMs. Figure 1 : An overview of how QuickVideo overlaps video decoding on CPU ( QuickCodec ) and prefill on GPU ( QuickPrefill ). QuickCodec concurrently processes intervals of the compressed video bit stream. QuickPrefill uses independent groups of frames, therefore it can begin prefill once the first frames are decoded, outputting carefully selected KV vectors. As QuickCodec loads frames synchronously, QuickPrefill can process the next prefill group immediately. This results in video decoding and prefill being almost enirely overlapped. To mitigate the computational overhead of long video VideoLLMs use extremely low frame sampling rates when processing long video inputs, instead of their native 1-2 FPS [ 4 , 7 ] . Frames are sampled as much as a minute apart during hour-long video understanding [ 49 , 8 ] . A minute gap between sampled frames can result in missing crucial video segments required for an understanding task. Low frame sampling rates also make fine-grained temporal and motion understanding impossible, as intervening frames are mostly removed [ 25 ] . Effective long video understanding thus requires loading and prefilling thousands of frames while preserving temporal continuity. Developing faster, more efficient VideoLLMs is critical for enabling comprehension of videos that span hours. Currently, video decoding and context prefilling are treated as disjoint and sequential stages in the VideoLLM pipeline. Moreover, video decoding is largely overlooked, despite contributing substantially to end-to-end latency. To remedy this, we introduce QuickVideo , a framework for faster, memory-efficient long video understanding. QuickVideo reduces the latency and resource requirements of these key bottlenecks in long video understanding. Our framework empowers fast video understanding on video inputs consisting of hundreds of thousands of frames, while maintaining the sampling rates required for fine-grained understanding. QuickVideo introduces three core contributions for accelerating long video understanding in VideoLLMs: (1) System-Level \rightarrow QuickCodec : a drop-in replacement video decoder designed for VideoLLMs. By redesigning video decoding for VideoLLM frame sampling, we achieve a 2-3x speedup compared to existing libraries when loading hour-long video inputs. (2) Algorithm-Level \rightarrow QuickPrefill : a group-based prefilling strategy combined with key-value (KV) cache pruning, which significantly reduces both computation time and memory usage during the prefilling stage, while incurring less than 3% accuracy degradation in most benchmarks. (3) Co-Design \rightarrow Overlapped Execution Scheme: the strategy tightly couples CPU-based QuickCodec and GPU-based QuickPrefill , enabling near-complete overlap to maximize efficiency. QuickVideo reduces the time to infer a 30 minute video input by more than 3x, from 69.7 seconds to only 20.0 seconds. The results demonstrate the effectiveness of our system-algorithm co-design. 2 Background We provide an overview of VideoLLM inference and key concepts in video processing. Although details vary, this background is broadly applicable to standard VideoLLM architectures and video standards. For clarity, we use “video decoding” to describe the process of decoding the compressed video into a tensor of video frames, and use “LLM decoding” to denote the process of auto-regressive decoding of a large language model. 2.1 VideoLLM Inference VideoLLMs must first decode a compressed video into a packed frame tensor before tokenization. The resulting raw frames are then passed through a visual encoder, which converts them into video tokens suitable for input to the LLM. Unlike text preprocessing, which relies on lightweight tokenizers, video decoding is inherently slow on both CPU and GPU due to its sequential nature [ 38 , 31 ] . Despite this, prior work in LLM video understanding has largely overlooked the latency incurred by this stage. Following preprocessing, the generation process of a VideoLLM consists of two stages: (1) Prefill , where both video and text tokens are processed to compute key-value (KV) caches for each transformer layer; and (2) LLM decoding , where

Read the complete text