QLIP : A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining

Kyle R. Chickering, Bangzheng Li, Muhao Chen · 2025

Read QLIP : A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining by Kyle R. Chickering, Bangzheng Li, Muhao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23004.txt.

Introduction

Kyle Chickering, Bangzheng Li, and Muhao Chen improve vision-language models without retraining in a 2025 paper. Most such models use the CLIP vision encoder. That encoder handles only fixed image sizes and can blur distinct images together.

Swapping the encoder normally forces costly retraining of the whole system. Their method, QLIP, adds a dynamic quadtree vision prior instead. The prior guides the model to spend detail where the image needs it. Two limits behind the encoder weakness are identified first. Gains appear without touching the trained weights. Readers learn a cheap upgrade path for large deployed models.

Catalog record

CREATORKyle R. Chickering, Bangzheng Li, Muhao Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23004

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QLIP : A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining Kyle R. Chickering, Bangzheng Li, Muhao Chen Abstract Multimodal Large Language Models (MLLMs) encode images into visual tokens, aligning visual and textual signals within a shared latent space to facilitate cross-modal representation learning. The CLIP model is a widely adopted foundational vision language model whose vision encoder has played a critical role in the development of MLLMs such as LLaVA. However, the CLIP vision encoder suffers from notable limitations including being constrained to only handling fixed input resolutions and a failure to produce separated embeddings for dissimilar images. Replacing the vision encoder of an existing model typically incurs substantial computational costs because such a change often necessitates retraining the entire model pipeline. In this work, we identify two factors which underlie the limitations of the CLIP vision encoder: mesoscopic bias and interpolation bias . To address these issues, we propose QLIP , a drop-in replacement for CLIP that can be seamlessly integrated with existing MLLMs with only a few lines of code and can enhance both coarse-grained and fine-grained visual understanding, without re-training. QLIP is designed around an image quadtree which replaces the standard uniform grid patches with a novel content aware patchification. Our experimental results demonstrate that QLIP improves the general visual question answering accuracy of the LLaVA-1.5 model series across various model sizes—without requiring retraining or fine-tuning of the full MLLM. Notably, QLIP boosts detailed understanding performance on the challenging V ∗ V^{*} benchmark by up to 13.6%. Figure 1: QLIP is a drop-in replacement for CLIP which allows models like LLaVA to perform inference on arbitrarily large images. In our experiments we find that vanilla LLaVA + QLIP gives +13.6% accuracy on the challenging V^{*} benchmark with no re-training or fine-tuning . The example in the figure above demonstrates an instance where CLIP cannot correctly get the answer because (a) in the cropped version of the image the person in question is not present, and (b) if we use a padded image the person will be too small to provide meaningful signal to model. 1 Introduction Multimodal Large Language Models (MLLMs) have shown impressive multi-modal question answering ability, yet recent work has highlighted a deficiency whereby these models struggle to answer questions about fine-grained visual details shi2024we ; wu2024v . MLLMs like the popular LLaVA family liu2024improved ; liu2023visual contain a vision encoder and visual projector which embed visual information into a sequence of tokens within a shared visual-linguistic embedding space before passing these tokens to an LLM. Recent work has demonstrated that, for many visual question answering (VQA) tasks, model performance is nearly unaffected by the removal of a high number of visual input tokens hu2024matryoshka ; li2025semantic ; sun2025lvpruning . It has also been shown that models like LLaVA overly rely on information from the vision encoder’s [CLS] token zhang2024cls to answer questions. Because the [CLS] token is a high-level encoding of an image’s content, reliance on this token does not aid in fine-grained visual analysis. We posit that this failure to perform satisfactorily on fine-grained VQA tasks is neither a deficiency in the training process of the MLLM nor a deficiency in the representations which can be encoded by the vision encoder. Prior works aimed at modifying the vision encoder or projector architectures have implicitly assumed that the failure mode is caused by CLIP itself, but this is only partially true. Li et al. li2025semantic show that the vanilla LLaVA architecture with the CLIP encoder is capable of much better VQA performance, but requires the “correct” tokens to be fed to the language model. Similarly, Li et al. li2024erroneous show that the information from the CLIP encoder is often sufficient for certain vision tasks or VQA, however, the models often do not adequately use the given information. We argue that the failures incurred while using the CLIP encoder can be attributed to two specific biases induced by the inductive priors which were implicitly assumed during the training of the CLIP encoder. Mesoscopic Bias occurs because CLIP uses a uniform grid-patchification (UGP) strategy dosovitskiy2020image ; radford2021learning and manifests as the model implicitly treating uniform grid cells at a specific image scale as the fundamental unit of semantic meaning. Interpolation Bias arises as a consequence of CLIP being trained with fixed positional embeddings on fixed-resolution images and prevents CLIP from handling high-resolution images. Previous work has focused on training a new vision encoder to replace CLIP guo2024llava ; liu2024improved ; luo2024feast ; shi2024we , but this requires re-training the entire MLLM, which is expensive and often not feasible for existing MLLM users. In this work, we take a minimally invasive approach and carefully reason through the consequences of updated vision priors. This lead us to a light-weight, content-aware, drop-in modification to the CLIP encoder which we name QLIP . QLIP empowers CLIP based MLLMs to automatically process arbitrary resolution input images, while adaptively scaling the number of input tokens based on the semantic content of the image. We find that reducing the number of input tokens has beneficial effects beyond reducing computation. We find that reducing image tokens in a reasoned way can reduce model hallucination and improve fine-grained VQA. To assess both the effectiveness and efficiency of QLIP , we apply it to the LLaVA-1.5 family of MLLMs for VQA. We emphasize fine-grained visual tasks like the challenging V^{*} benchmark wu2024v . Our method achieves a 13.6% improvement on V^{*} , reduces hallucination rates as measured by the POPE F1 score li2023evaluating by 5.2, and yields notable improvements across other multi-modal benchmarks including MME fu2023mme and RealWorld-QA xai2024realworldqa . We accomplish this using two novel strategies. First, to address the mesoscopic bias we introduce a non-uniform patchification scheme based on image quadtrees hunter1979operations . Our patchification procedure is adaptive, tunable, and training-free , and implicitly treats semantically similar regions of the image as the fundamental unit of semantic meaning instead of UGP. Second, to address the interpolation bias, we train a small MLP network to interpolate the fixed positional CLIP embeddings while maintaining usable positional signals for downstream models. This small interpolation network requires little training yet is both highly effective and generalizable. Our key contributions are as follows: 1. We identify two fundamental biases in the CLIP vision encoder, i.e. mesoscopic bias and interpolation bia, and propose quantitative measures of both. 2. We introduce QLIP , a lightweight, drop-in modification for CLIP that supports arbitrary image resolutions and adaptively scales the number of the image tokens based on image content. QLIP directly mitigates the aforementioned biases without modifying the original encoder weights or requiring expensive re-training of the MLLM. 3. We empirically validate the effectiveness of QLIP by integrating it into the LLaVA model family liu2024improved ; liu2023visual and demonstrating substantial performance improvements. Our results are achieved without any supervised fine-tuning or re-training of the model backbone. For the challenging V^{*} benchmark, we achieve a significant improvement of +13.6% accuracy using LLaVA 13B with QLIP , outperforming the previous SoTA CLIP-based LLaVA results by +3.1% shi2024we . 2 Why CLIP Fails at Higher Resolutions The CLIP vision encoder is trained at a fixed input resolution using learned absolute positional encodings radford2021learning .

Read the complete text