FILA: Fine-Grained Vision Language Models
Shiding Zhu, Wenhui Dong, Jun Song, Yingbo Wang, Yanan Guo, Bo Zheng · 2024
Read FILA: Fine-Grained Vision Language Models by Shiding Zhu, Wenhui Dong, Jun Song, Yingbo Wang, Yanan Guo, Bo Zheng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08378.txt.
Introduction
Vision-language models read images best at low resolution, so the usual trick for detailed pictures is to slice them into tiles, which cuts objects and text apart at the seams. Shiding Zhu and co-authors propose HyViLM, a model built to swallow high-resolution images whole. Its hybrid encoder processes each tile while also exchanging information with global features of the full image, keeping context intact during encoding.
A fusion strategy then combines features from several layers of the encoder, using detail wherever it lives. Against comparable multimodal models, HyViLM won nine of ten benchmark tasks, with its largest gains on text-heavy image questions: about 9.6 percent better on TextVQA and 6.9 percent on DocVQA. The authors present the design as the fix for the semantic breaks that tiling introduces.
Catalog record
| CREATOR | Shiding Zhu, Wenhui Dong, Jun Song, Yingbo Wang, Yanan Guo, Bo Zheng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FILA: Fine-Grained Vision Language Models Shiding Zhu, Wenhui Dong, Jun Song, Yingbo Wang, Yanan Guo, Bo Zheng Abstract Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. While this method enables MLLMs to process high-resolution images within input constraints, it often causes semantic fragmentation by truncating objects and connected areas, leading to degraded model performance, particularly for fine-grained details and irregularly shaped objects. To address this limitation, we propose FILA ( F ine-Grained V i sion La nguage Model), a novel framework designed to process images of high resolution while retaining the overall context during encoding. Specifically, we: (i) Introduce a Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model’s ability to encode high-resolution images. (ii) Propose an effective feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our FILA outperforms existing MLLMs in nine out of ten tasks. Specifically, FILA achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task. 1 Introduction (a) Figure 1: (a) An overall performance comparison between FILA and other existing MLLMs. (b) and (c) are examples of image fragments. The solid red line in the figures represents the cropping boundary. LLaVA-NeXT provides correct answers when pivotal information remains within the cropping boundaries. However, even slight overlaps with the boundaries cause it to misidentify the information. Inspired by the remarkable achievements of large language models (LLMs) ( Zhang et al., 2022 ; Brown et al., 2020 ; Touvron et al., 2023 ) , the development of multimodal large language models (MLLMs) ( Alayrac et al., 2022 ; Bai et al., 2023 ; Chen et al., 2024b ; Li et al., 2023 ; Zhu et al., 2023 ; Liu et al., 2024a ) is advancing at a rapid pace. Researchers have directed substantial efforts towards broadening the capabilities of LLMs to encompass additional modalities, resulting in significant breakthroughs in assorted vision-language tasks. Nevertheless, the challenge of fine-grained visual recognition is not adequately addressed. This shortcoming arises partly from the constraints inherent in pre-trained visual encoders ( Zhai et al., 2023 ; Radford et al., 2021 ) , which often require MLLMs to process images at relatively low resolutions, typically around 224\times 224 ( Zhu et al., 2023 ; Ye et al., 2024 ) or 336\times 336 ( Liu et al., 2024b ) . This limitation hinders the precise identification of fine details and contextual elements within images, resulting in less accurate vision understanding and reasoning. To solve this limitation, recent endeavors are directed toward enabling MLLMs to handle images of high resolution. One approach involves directly utilizing a visual encoder designed to process higher-resolution images. However, this approach necessitates the training of a high-quality visual encoder, which requires significant computational resources, including high-quality, high-resolution training datasets. An alternative effective strategy is dynamic cropping ( Ye et al., 2023 ; Li et al., 2024b ; Xu et al., 2024 ; Hu et al., 2024 ; Liu et al., 2024d ; Liu et al., 2024b ) , where the high-resolution image is partitioned into a set of lower-resolution sub-images using adaptive algorithms. These segmented sub-images are subsequently processed by a visual encoder that has been previously trained with lower-resolution image data, thereby augmenting the encoder’s ability to capture finer details while ensuring that maintaining computational efficiency. Nevertheless, this cropping method can result in image fragmentation, thereby altering the original spatial context and disrupting positional associations within the image. This becomes particularly problematic when essential information is located at the edges of these cropped segments, as it poses challenges to encoding such information accurately. For instance, as depicted in Figure 1 when the pivotal information does not intersect with the cropping boundaries, the model LLaVA-NeXT ( Liu et al., 2024b ) provides the correct answer. However, with a slight alteration in the image such that the pivotal information intersects with the cropping boundaries, LLaVA-NeXT fails to identify it correctly. To alleviate this effect, in this paper, we propose FILA enhancing fine-grained recognition. First, we design a visual encoder, Hybrid Encoder, under the dynamic cropping strategy to address the issue of image fragmentation. Hybrid Encoder can acquire comprehensive global information to enhance the encoding of each sub-image. Furthermore, we propose a different way of visual feature interaction called ConvNeXt-ViT Deep Fusion Module (CVFM), which fully utilizes features at different levels for mutual complementarity, achieving optimal visual encoding. Figure 2 demonstrates the difference between our approach and previous methods. Previous methods include Dynamic Resolution, Mini-Gemini ( Li et al., 2024a ) , and Channel-wise Concat. Dynamic Resolution simply involves cropping the image and feeding it into CLIP-ViT ( Dosovitskiy, 2020 ) . Mini-Gemini interacts the low-resolution CLIP-ViT features with a high-resolution auxiliary branch at the last layer, mainly using Cross Attention, while some methods utilize Channel-wise Concat. The innovation of our method lies in our interaction between two high-resolution branches and the novel idea of deep fusion, which successfully addresses the issue of image fragmentation. To verify the effectiveness of our proposed method, we conducted extensive experiments on ten vision-language tasks, some of which include fine-grained recognition of high-resolution images, such as TextVQA ( Singh et al., 2019 ) and DocVQA ( Mathew et al., 2021b ) . The experimental results show that our method outperforms existing MLLMs on 9 out of 10 Vision-Language tasks. Specifically, it surpasses LLaVA-NeXT by 9.6% on TextVQA and by 6.9% on DocVQA. In summary, our contributions are three folds: • We propose a new visual encoder, Hybrid Encoder, to solve the problem of image fragmentation under dynamic cropping strategies. Hybrid Encoder can utilize comprehensive global information to enhance sub-image encoding, improving the model’s upper limit in handling high-resolution images. • We design a novel visual feature fusion module CVFM, cleverly utilizing information at different levels for mutual complementarity. • Based on our strategy, we developed the FILA model, a powerful MLLM that outperforms existing models on multiple benchmarks, demonstrating the effectiveness of our approach. Figure 2: An Overview of High-Resolution Methods. This Figure illustrates the differences between our method and previous methods. 2 Related Work 2.1 Large Vision-Language Model Driven by the tremendous success of large language models (LLMs), there has been growing interest in building end-to-end multimodal large language models. Specifically, most existing MLLMs adopt a modular structure, utilizing an intermediate network to project visual features into the word embedding space of an LLM. Then, the LLM completes various vision-language (VL) tasks in an autoregressive manner. The intermediate networks of existing MLLMs can generally be divided into two types: (i) learned queries, such as perceiver resamplers ( Alayrac et al., 2022 ) or Q-Formers ( Li et al., 2023 ) , which use fixed queries to capture features through cross-attention; and (ii) MLP modules, like those in the LLaVA series ( Liu et al., 2024a ; Liu et al.,