Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings

Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, Aixin Sun · 2025

Read Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings by Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, Aixin Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04997.txt.

Introduction

Searching scanned documents by their look, not just their text, has grown powerful with models that encode each page as hundreds of patch embeddings. The cost is memory: such stores swell fast. Ma and ten co-authors study how to shrink those embeddings with the least loss of search quality. They test two trims: dropping patches, called pruning, and fusing them, called merging.

Random pruning oddly beat smarter pruning methods, yet all pruning fell short, since a page's key patches depend on the question asked. Merging worked far better. Their tuned version, Light-ColPali and Light-ColQwen2, keeps about ninety-eight percent of search quality with about twelve percent of the memory. At under three percent of the memory, it still keeps about ninety-five percent. The authors offer the result as a baseline for leaner document search.

Catalog record

CREATORYubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, Aixin Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, Aixin Sun Abstract Despite the strong performance of ColPali/ColQwen2 in Visualized Document Retrieval (VDR), it encodes each page into multiple patch-level embeddings and leads to excessive memory usage. This empirical study investigates methods to reduce patch embeddings per page at minimum performance degradation. We evaluate two token-reduction strategies: token pruning and token merging . Regarding token pruning, we surprisingly observe that a simple random strategy outperforms other sophisticated pruning methods, though still far from satisfactory. Further analysis reveals that pruning is inherently unsuitable for VDR as it requires removing certain page embeddings without query-specific information. Turning to token merging (more suitable for VDR), we search for the optimal combinations of merging strategy across three dimensions and develop Light-ColPali/ColQwen2. It maintains 98.2% of retrieval performance with only 11.8% of original memory usage, and preserves 94.6% effectiveness at 2.8% memory footprint. We expect our empirical findings and resulting Light-ColPali/ColQwen2 offer valuable insights and establish a competitive baseline for future research towards efficient VDR. 1 1 footnotetext: Corresponding Author 1 Introduction Visualized Document Retrieval (VDR) matches queries to relevant documents by leveraging their visual representations. Unlike conventional retrieval systems, where raw text must be parsed before indexing, VDR captures documents as images (screenshots) and encodes them into embeddings using Large Vision-Language Models (LVLMs). This approach preserves layout structures and visual elements, enabling retrieval in a what-you-see-is-what-you-get manner. As a result, VDR achieves superior retrieval accuracy and demonstrates strong potential across various applications Cho et al. 2024 ; Chen et al. 2025 . Figure 1: Top : The relative memory consumptions for embedding storage of different VDRs. Our simple yet effective approach, Light-ColPali/ColQwen2, retains most of the performance but with significantly reduced memory cost. Bottom : The diagram of VDR equipped with ColPali/ColQwen2 retriever. It encodes each page into N_{p} patch-level embeddings and thus incurs prohibitive memory cost. The state-of-the-art visualized document retriever, ColPali/ColQwen2 Faysse et al. 2025 , represents a significant advancement in this field. As shown in Figure 1 , ColPali/ColQwen2 encodes each document page as N_{p} patch-level embeddings during the offline stage and saves them for online computation. While the excessive number of patch embeddings enables the perceiving of fine-grained details (which is particularly important for document-related images), it introduces substantial memory footprints and computational overhead in both offline indexing storage and online similarity computation. For example, a medium-sized document with 50 pages requires about 10 MB memory for embedding storage * * * ColQwen2 divides each page into 768 tokens, each represented by a 128-dimensional vector. Stored as 16-bit floats, it requires 50*768*128*16 bits = 9.6 MB per document. . This substantial memory footprint presents a bottleneck for scalability and practical deployment of VDR systems under real-world scenarios. In this work, we present an in-depth analysis of the storage-efficient visualized document retriever, exploring how to reduce each page’s patch embedding number with minimal performance degradation . We consider two common token-reduction approaches, i.e., token pruning Chen et al. 2024 and token merging Clavié et al. 2024 , respectively. In terms of token pruning, we investigate multiple pruning strategies in Section 4 and aim to retain only the high-informative patch embeddings. Even though token pruning works to some extent, it can not reduce the embedding numbers by orders of magnitude without significant performance drops. More embarrassingly, we observe that the most simple strategy, i.e., random pruning, performs better than other carefully designed strategies. A deeper analysis of this observation reveals that (1) the informativeness of patch embedding is highly conditioned on the queries, which are unknown and unpredictable during the offline indexing stage. (2) the patch embeddings can be grouped and, accordingly, are prone to be dropped by the group under some specific criteria. The above two reasons make it impractical to decide which embeddings should be pruned without access to the queries. Therefore, we claim that pruning-related strategies are inappropriate under VDR settings. In Section 5 , we investigate token merging strategies across three critical dimensions: (1) merging approaches, (2) fine-tuning applicability, and (3) merging locations. Our analysis reveals that similarity-based clustering marginally surpasses spatial-oriented pooling in effectiveness, and resource-efficient fine-tuning (about 72 A100-GPU hours) significantly mitigates the performance gap between retrievers with and without merging. Additionally, we observe that late-stage merging ( i.e., at the final layer of ColPali/ColQwen2) optimally preserves information and minimizes performance degradation. Building upon these insights, we establish a simple yet effective baseline, named Light-ColPali/ColQwen2, for patch-level embedding reduction in VDR systems. Comprehensive evaluations across three benchmarks Faysse et al. 2025 ; Yu et al. 2024 ; Ma et al. 2024b demonstrate that Light-ColPali/ColQwen2 approach maintains comparable performance while achieving orders-of-magnitude reduction in patch storage requirements. Notably, as illustrated in Figure 1 , it retains 98.2% of NDCG@5 scores with only 11.8% of original memory footprint and maintains 94.6% effectiveness at just 2.8% memory footprint. 2 Related Work Visualized Document Retriever shares a similar architecture with text-based dense retrievers Karpukhin et al. 2020 ; Khattab and Zaharia 2020 , but leverages LVLMs Wang et al. 2024 ; Beyer et al. 2024 for OCR-free document understanding. It primarily divides into two approaches: (1) Page-level embedding retrievers (DSE; Ma et al. 2024a ) encode entire pages and queries into single embeddings; (2) Patch-level embedding retrievers (ColPali/ColQwen2; Faysse et al. 2025 ) generate multiple patch-level embeddings per page and token-level embeddings per query. While patch-level retrievers offer finer granularity and superior performance, they demand prohibitive computational resources for both offline indexing and online retrieval. This work addresses this limitation by developing methods to reduce embedding numbers in patch-level retrievers. Token-reduction about LVLM has been intensively explored to improve LVLMs’ generation ( i.e., next-token prediction) efficiency. Current approaches fall into three categories: (1) Pruning strategies Liang et al. 2022 ; Chen et al. 2024 ; Xing et al. 2024 ; Zhang et al. 2025 eliminate low-information tokens based on importance ranking; (2) Merging strategies Kong et al. 2022 ; Bolya et al. 2023 ; Li et al. 2024b combine similar tokens into compressed embeddings; (3) Hybrid strategies Shang et al. 2024 ; Yang et al. 2024 ; Wu 2025 integrate pruning and merging, usually by preserving high-informative tokens while merging redundant ones. In LVLM generation, the focus is on minimizing response latency and FLOPs given specific instructions. In document retrieval, the goal is to reduce the memory footprint of embeddings, without query access but with less concern for latency or FLOPs during indexing. These distinctions relax constraints on token merging, enabling late-stage compression and more computationally-intensive merging strategies. However, the absence of queries precludes query-conditioned pruning or merging approaches. Lightweight Document Retriever has been explored to address the challenge of large-scale embeddings with two orthogonal approaches: (1) Dimension Reduction . ColBERTv2 Santhanam et al. 2022 employs product quantization Jégou et al. 2011 to reduce the size of each embedding from 768 to 128 dimensions. This design is inherited by ColPali Faysse et al. 2025 with a simpler projection layer. (2) Token Reduction : Clavié et al. 2024 introduces the concept of TokenPooling and explores merging strategies for

Read the complete text