B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens

Zhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, Kun Hu · 2024

Read B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens by Zhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, Kun Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09919.txt.

Introduction

Zhuqiang Lu and co-authors present B-VLLM, a vision language model that handles long videos in a paper from 2024. Video frames turn into visual tokens, and long videos make millions of them. That flood can overflow the model's context and slow it to a crawl.

Past fixes either kept few frames, losing time detail, or shrank each frame, losing space detail. B-VLLM balances space and time instead of trading one for the other. Tokens are budgeted so scenes keep their layout and motion stays readable. The model can then chat about both images and text in one stream. Long-video understanding matters for surveillance, sports, and film search alike.

Catalog record

CREATORZhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, Kun Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens Zhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, Kun Hu Abstract Recently, Vision Large Language Models (VLLMs) with integrated vision encoders have shown promising performance in vision understanding. They encode visual content into sequences of visual tokens, enabling joint processing of visual and textual data. However, understanding videos, especially long videos, remains a challenge as the rapid growth of visual tokens during video encoding risks exceeding VLLMs’ context window length and significantly escalates computational cost. To restrict the number of visual tokens, existing VLLMs either: (1) uniformly downsample videos into a fixed number of frames or (2) reducing the number of visual tokens encoded from each frame. We argue that the former neglects temporal dynamics in videos, while the latter fails to preserve spatial details within individual frame. In this work, we propose Balanced-VLLM (B-VLLM), a novel VLLM framework designed to model task relevant spatio-temporal cues, while restricting the number of visual tokens within the VLLM’s context window length. Central to our framework is a text-conditioned adaptive frame selection module that dynamically identifies task-relevant frames, which are further de-duplicated with a temporal frame token merging strategy. The visual tokens of these frames then undergo spatial token sampling and an optional spatial token merging strategy for granular control against the token budget. Experiments demonstrate the effectiveness of B-VLLM in balancing the number of frames and visual tokens, moreover, our proposed method introduces 10% performance gain on MVBench. Code is available at https://github.com/zhuqiangLu/B-VLLM.git . 2 2 footnotetext: Corresponding authors. 1 Introduction (a) Context window limit. (b) Key-hints missed in sampling. (c) Video LLM performance on image-based spatial-only benchmarks. Figure 1 : Illustration of the limitations of existing VLLMs. Large Language Models (LLMs) have demonstrated remarkable capability and generalizability in natural language understanding [ 4 , 56 , 51 , 11 ] . Recently, there have been several notable efforts to extend LLMs into Vision Large Language Models (VLLMs) for visual understanding [ 2 , 33 , 62 ] . These efforts primarily aim to adapt LLMs’ contextual reasoning capabilities to multimodal settings like video captioning and visual question answering, enabling them to generate accurate textual responses based on visual and textual context provided by users. Unlike the extension of vision models to downstream tasks [ 19 , 57 , 37 , 61 , 8 ] , bridging the modality gap between textual and visual contexts poses significant challenges [ 33 ] . Existing VLLMs typically use pretrained Vision Transformers (ViTs) [ 10 ] to encode images into sequences of visual tokens. These tokens are subsequently integrated with text token sequences as input to the VLLMs [ 33 , 32 ] . This approach compresses spatial contexts from images into 1D sequences and has demonstrated its effectiveness across various computer vision tasks [ 14 ] . However, these VLLMs are primarily designed for single-image scenarios. As illustrated in Figure 1(a) , extending them to video data often compromises performance, as the number of visual tokens exceeds the length limit of the VLLM’s context window. To address this issue, methods like LLaMA-VID [ 30 ] , MovieChat [ 50 ] and LongVLM [ 54 ] reduce the number of visual tokens to encode each frame. Alternatively, methods such as VideoLLaMA2 [ 9 ] , VideoLLaMA [ 58 ] and VideoChat2 [ 27 ] sample a fixed number of frames uniformly from a video to prevent an overload of visual tokens. While being effective in controlling the number of visual tokens for video processing, these methods struggle to balance spatio-temporal cues. As depicted in Figure 1(b) , VLLMs using uniform frame sampling tend to underperform on temporally relevant tasks. As shown in Figure 1(c) , methods that reduce the number of visual tokens per frame often perform poorly on spatially relevant tasks. We argue that this is due to a spatio-temporal token imbalance: reducing frame-level visual tokens causes temporal cues to dominate, whereas uniform frame sampling leads to spatial cues to be overshadowed. To address these issues, we propose Balanced-VLLM (B-VLLM), a novel VLLM framework to balance spatio-temporal cues in video data conditioned by task-related text prompts. B-VLLM processes each frame with both coarse-grained and fine-grained visual tokens. A learnable frame selection module leverages coarse-grained tokens to identify the frames most relevant to the video understanding task, guided by the text prompt. The fine-grained visual tokens of these selected frames are then processed by a spatial visual token sampling module to retrieve those most task-relevant ones. For granular control over the final number of generated visual tokens, we introduce an iterative spatial token merging strategy that averages similar visual tokens until the expected number of tokens is reached. Finally, the resulting visual tokens are mapped into the LLM feature space and integrated with the sequence of textual tokens encoded from the text prompt before being fed into the LLM. Our proposed method achieves SOTA performance on various benchmarks. Notably, our method introduces up to 10% performance improvement on MVBench [ 27 ] The main contributions are summarized as follows: • We propose B-VLLM, a novel Vision LLM framework that dynamically balances spatio-temporal tokens for video understanding, especially for long videos. • We devise an adaptive token selection strategy by leveraging both coarse-grained and fine-grained tokens. • We undertake comprehensive experiments to show that B-VLLM is effective in different VLLM architectures for video understanding. • We demonstrate that the B-VLLM framework can be generalized to different datasets and VLLM settings. 2 Related Work 2.1 Vision Large Language Models Vision Large Language Models (VLLMs) extend existing Large Language Models (LLMs) to the domain of visual understanding through visual-instruction tuning [ 33 ] . Visual content is encoded as sequences of visual tokens using a pretrained CLIP [ 44 ] vision encoder. These visual tokens are then projected into the LLM hyperspace through a trainable MLP layer [ 33 , 30 , 9 , 31 ] . However, in multi-image or video scenarios, the number of vision tokens often exceeds the VLLM’s context window limit, leading to performance degrading and high computation costs. To address this, Open-Flamingo [ 2 ] , MovieChat [ 50 ] and BLIP2 [ 26 ] reduce the number of vision tokens extracted from each frame by using transformer-based token selection networks, such as Q-Former [ 26 ] and Perceptive Resampler [ 2 ] . However, the number of vision tokens is proportional to the number of input images or frames, creating the risk of an overwhelming number of tokens when processing long videos. LLaMA-VID [ 30 ] addresses this by encoding each frame into only two visual tokens to enable hour-long video processing; however, this extreme compression rate limits LLaMA-VID’s ability to capture finer spatial details in each frame. Another common approach to controlling the number of visual tokens is to uniformly sample frames from a video [ 33 , 31 , 32 , 53 , 9 ] . However, uniformly frame sampling may overlook important frames, especially when keyframes of interests are not evenly distributed throughout the video. Compared to existing studies, we propose a vision LLM framework empowered by an adaptive, text-conditioned frame selection mechanism that dynamically

Read the complete text