LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong · 2026

Read LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs by Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17260.txt.

Introduction

Kim and colleagues present LiteFrame, a compact vision encoder that lets video language models scale to far more frames. Their compressed token distillation trains a small student to predict dense, compressed views from a large teacher, skipping redundant per-frame work. Paired with language-model adaptation, it cuts end-to-end latency 35 percent against InternVL3-8B while handling eight times more frames and lifting average accuracy.

Catalog record

CREATORJihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52738
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong Abstract The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on “post-hoc” token reduction—reducing visual tokens after feature extraction to alleviate the LLM’s computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame , a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier—compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8 × \times more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets. Project Page: jjihwan.github.io/projects/LiteFrame \uselogo 1 Introduction Figure 1: We propose LiteFrame , a lightweight video encoder that reduces inefficiencies from both the LLM and the ViT. (a) Standard Video LLMs ( 43 ) are bottlenecked by the LLM’s quadratic complexity, strictly limiting context length to \sim 64 frames. (b) Post-hoc reduction alleviates LLM’s burden, enabling more frames, yet ironically shifts the bottleneck to the ViT, causing latency to explode. (c) Our approach (LiteFrame) resolves both inefficiencies, enabling 12.7 \times faster LLM prefilling and 5.3 \times faster ViT encoding at 64 frames compared to the InternVL3-8B baseline. Modern Multimodal LLMs (MLLMs) ( 43 ; 22 ; 32 ; 15 ; 20 ) have achieved remarkable progress in recent years in video understanding, parsing complex temporal dynamics for captioning ( 7 ) , question answering ( 9 ; 42 ) , and reasoning ( 10 ) . Despite these strong capabilities, there remains a fundamental scaling problem when handling long-form video within the current paradigm—the computational cost of processing spatio-temporal video data grows prohibitively with increasing frame counts. To understand why this is, we first note that these models all typically follow a very similar multi-stage architecture consisting of an image encoder (e.g. Vision Transformer; 6 ) that processes a video frame by frame, an alignment projector, and an LLM that reasons over the interleaved visual and text tokens. Therefore, with each additional input frame, the computational cost increases due to processing demands in both the vision encoder and the LLM. Existing works that try to alleviate this computational burden have largely focused on the LLM, attributing the primary bottleneck to the quadratic complexity of self-attention over an increasing number of visual tokens. Consequently, the dominant solution has been an “extract-and-reduce” paradigm, which maintains the frozen image encoder for frame-level feature extraction, and leverages post-hoc token reduction strategies ( Figure 1 (b))—either spatially ( 24 ; 38 ; 33 ) , spatio-temporally ( 29 ; 14 ; 26 ; 25 ) , or via query-guided pruning ( 2 ; 36 ; 27 ; 37 ) —before feeding them to the LLM. We show that this class of methods ignores the cost of the per-frame feature extraction, which while seemingly lightweight, becomes cumulatively expensive. Specifically, our preliminary analysis in Section 3 reveals that while aggressive post-hoc token reduction (e.g., 16\times ) alleviates the LLM overhead, as the LLM compute decreases, the computational burden of the visual encoder begins to dominate. This remaining bottleneck prohibitively sets a floor for the achievable end-to-end inference efficiency. As illustrated in Figure 1 (b), once post-hoc token reduction is effectively applied, the vision encoder’s latency becomes the new bottleneck as frame counts increase. Hence, unlocking the next generation of efficient MLLMs for long-video understanding requires a holistic approach that simultaneously optimizes both visual encoding and language model efficiency. To this end, we introduce LiteFrame, a lightweight, efficient video encoder designed to reduce per-frame compute with a minimal decrease in video understanding accuracy. To achieve this, we propose Compressed Token Distillation (CTD), a novel strategy for training a compute-efficient, token-compressive encoder from a pretrained teacher image encoder. Specifically, CTD directly aligns the student with an information-dense, spatio-temporally compressed teacher output. Furthermore, we design the student encoder architecture to explicitly reduce spatio-temporal redundancies across frames. When coupled with a lightweight Language Model Adaptation (LMA) stage (adapting the new encoder with the LLM), LiteFrame allows Video LLMs to achieve a new latency-accuracy Pareto frontier for video understanding. As illustrated in Figure 2 , our model delivers superior accuracy with remarkably low latency when compared to existing baselines. Specifically, LiteFrame significantly outperforms the InternVL3-8B by processing 8\times more frames with a 35% reduction in end-to-end latency, while using only 87M parameters (vs. 304M for the teacher). To summarize our contributions: • We identify a critical scaling blindspot in current efficient Video LLM paradigms: while post-hoc token reduction effectively alleviates LLM computational costs, the vision encoder becomes the new latency bottleneck, preventing further efficient scaling to long videos. • We propose LiteFrame, an efficient video encoder that resolves this bottleneck shift by integrating token compression directly within a lightweight visual backbone. • We introduce Compressed Token Distillation (CTD), a novel training framework for maximizing the transfer of spatio-temporally dense information from a teacher to a compact student. • Extensive experiments demonstrate that LiteFrame redefines the performance-latency trade-off. Our approach achieves a 1.53\times acceleration in end-to-end inference compared to the InternVL3-8B teacher, while processing 8\times more frames and outperforming the baselines on multiple video understanding tasks. Figure 2: The pareto frontier of video understanding efficiency. We illustrate the trade-off between average accuracy on four video benchmarks (Video-MME w/ and w/o subtitles, MLVU, and LongVideoBench) and end-to-end latency including vision encoding and LLM prefilling. Our proposed post-hoc primitive, Weighted Average Pooling (WAP, red triangles), and our efficient video encoder, LiteFrame (red stars), push the efficiency Pareto frontier, achieving superior accuracy compared to the teacher model (InternVL3, black dashed). Existing post-hoc methods (color dashed) fail to improve the trade-off as they neglect the encoder latency bottleneck. Note that the x-axis (latency) is log-scaled. 2 Related work Post-hoc token reduction. The predominant method for making MLLMs efficient is the “extract-and-reduce” paradigm: applying post-hoc token reduction after heavy, pre-trained vision encoders extract dense features, aiming to reduce the cost attributed to the LLM’s quadratic complexity. Early approaches focused on spatial redundancy within individual images, using adaptive selection or merging ( 24 ; 38 ) . More recent efforts extend this to the temporal dimension for video inputs via dynamic pruning or holistic merging ( 26 ; 29 ; 14 ; 25 ) . While these post-hoc methods reduce the computational burden on the LLM, they remain inefficient for long-form video understanding (hundreds or thousands of frames) because they miss a critical scaling bottleneck. Because these methods rely on a heavy, frozen encoder to process every frame prior to compression, the latency bottleneck shifts

Read the complete text