SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models

Thong Nguyen, Yibin Lei, Jia-Huei Ju, Andrew Yates · 2025

Read SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models by Thong Nguyen, Yibin Lei, Jia-Huei Ju, Andrew Yates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15432.txt.

Introduction

Thong Nguyen leads a team that rethinks how to search scanned documents. The usual way trains special models to match text queries to document images. The team try a simpler path that needs no such training. A vision language model first writes a full text account of each document image. A standard text search model then works on those accounts.

On a known test set, their path scores higher than the best special trained model. It also scales better to large sets and covers more languages. Their study shows current vision models catch fine detail in both words and layout. The result sets a strong no training base line for future search tools.

Catalog record

CREATORThong Nguyen, Yibin Lei, Jia-Huei Ju, Andrew Yates
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models Thong Nguyen, Yibin Lei, Jia-Huei Ju, Andrew Yates Abstract Visual Document Retrieval (VDR) typically operates as text-to-image retrieval using specialized bi-encoders trained to directly embed document images. We revisit a zero-shot generate-and-encode pipeline: a vision–language model first produces a detailed textual description of each document image, which is then embedded by a standard text encoder. On the ViDoRe-v2 benchmark, the method reaches 63.4% nDCG@5, surpassing the strongest specialised multi-vector visual document encoder. It also scales better to large collections and offers broader multilingual coverage. Analysis shows that modern vision–language models capture complex textual and visual cues with sufficient granularity to act as a reusable semantic proxy. By offloading modality alignment to pretrained vision–language models, our approach removes the need for computationally intensive text-image contrastive training and establishes a strong zero-shot baseline for future VDR systems. Our code is available for reproduction at: thongnt99/serval 1 Introduction and Related Work Real - world documents originate from diverse sources, spanning the public web to private enterprise repositories, and appear in many formats, including plain text, figures, graphs, and tables. Document retrieval bridges human or artificial agents to the most relevant information, enabling informed decision - making and knowledge synthesis. Figure 1: nDCG@5 for zero-shot Visual Document Retrieval using VLMs and text encoders of varying scales. Despite no task-specific training, our zero-shot method could compete with end-to-end models explicitly trained for VDR on large-scale text-(document) image datasets. Over several decades the field has been dominated by text - centric retrieval methods. Classic approaches such as BM25 Robertson and Walker 1994 rely purely on lexical matching between queries and documents. The recent advent of deep learning, and especially transformer architectures such as BERT, has shifted the focus to context - aware neural retrieval. Today a rich ecosystem of neural paradigms is available, from sparse and dense first - stage encoders Wang et al. 2022 ; Wang et al. 2024 ; Formal et al. 2021 ; Lassance et al. 2024 ; Nguyen et al. 2023 to cross - encoders for re - ranking Nogueira et al. 2020 ; MacAvaney et al. 2019 ; Reimers and Gurevych 2019 . Trained on large human - and machine - annotated corpora, these models achieve state - of - the - art performance on in - domain Bajaj et al. 2016 , out - of - domain Thakur et al. 2021 , and multilingual Zhang et al. 2023 ; Enevoldsen et al. 2025 ; Zhang et al. 2021 benchmarks. Text - only retrieval overlooks other important visual elements embedded in documents Xu et al. 2020 . Visual Document Retrieval (VDR) tackles this gap by jointly encoding textual and visual elements rendered as images. Ma et al. 2024 formulate VDR as text - to - image retrieval by rasterizing each page, while CoPali introduces a multi - vector encoder together with the ViDoRe benchmark Faysse et al. 2024 . To spur further progress, ViDoRe - v2 Macé et al. 2025 raises the bar with human - verified, multilingual queries that are substantially more challenging. To close the modality gap, state-of-the-art VDR systems typically rely on expensive contrastive training over high-quality text and visual document pairs. However, we hypothesize that advanced vision-language models (VLMs) can bridge this gap by effectively describing visual elements in language. Therefore, we reformulate the end-to-end paradigm and revisit a simple zero-shot alternative that decouples the problem into two independent sub-tasks: (i) document-description generation, handled by a VLM, and (ii) text encoding, handled by a conventional pretrained text encoder. This modular design enables us to plug in best-in-class components and leverage the growing number of high-quality VLMs and text encoders. For description generation, we exploit recent VLMs, including Qwen2.5VL Bai et al. 2025 and InternVL 3 Zhu et al. 2025 , that excel at visual understanding. For text encoding we leverage robust open - source multilingual encoders Wang et al. 2022 ; Wang et al. 2024 ; Meng et al. 2024 ; Yang et al. 2025 usually equipped with instruction - following capabilities Weller et al. 2024 . Evaluated on the nine ViDoRe - v2 tasks and MIRACL-VISION benchmark, our zero-shot generate-and-encode approach matches or even surpasses state-of-the-art supervised multi - vector baselines (as shown in Figure 1 ), despite using no VDR - specific training data. Our analysis produces three key insights: • Recent vision–language models (VLMs) can accurately describe visual elements embedded in documents–figures, graphs, and tables–enabling effective visual document retrieval from their generated descriptions. • Scaling both VLMs and text encoders improves retrieval performance, but scaling/improving the text encoder yields better gains. Large (32B / 72B) VLMs paired with strong text encoders achieve the best scores, while even 2B–7B models already surpass most supervised end-to-end models on ViDoRe-v2. • Supervised end-to-end VDR models perform well on English, but they lag behind on multilingual and cross-lingual tasks, highlighting potential room for future improvement. Because description generation is executed offline during indexing, online latency remains similar to end-to-end VDR approaches. Smaller VLMs (2B–7B) offer an attractive speed–accuracy trade - off, providing competitive accuracy while reducing offline document preprocessing cost. 2 Zero-shot Visual Document Retrieval Figure 2: Zero-shot VDR using VLM-generated document descriptions and a pretrained text encoder. The workflow for our zero-shot VDR approach is illustrated in Figure 2 . First, a vision-language model (VLM) is used to generate a detailed description of the visual document. Using the prompt shown in Prompt 2 , we instruct the VLM to begin with an overall summary of the content depicted in the image, followed by a comprehensive list of details, including any extracted text and numerical values. The entire description is generated in a single step, bypassing intermediate procedures such as layout detection or document chunking used in CoPali Faysse et al. 2024 . An example of a generated description from InternVL3 is shown in Figure 3 . Additional examples covering various complex document types (e.g., graphs, tables, diagrams) are provided in the Appendix A . Prompt. 2.1: Description generation with VLMs Provide a comprehensive description of the document in the image in English. Begin with a summary, then follow with details. Extract all visible text and numerical values from the document. In the second step, any off-the-shelf text encoder can be used to encode both queries and the generated document descriptions. The goal of this encoder is to bridge the semantic gap between queries and documents by mapping them into a shared semantic space where relevant pairs are positioned closely. It is also important for the encoder to support multilingual and cross-lingual retrieval, as the evaluation will include non-English data. For large-scale document collections, document descriptions and their embeddings can be pre-computed and indexed offline in a database. 3 Experimental Setup We evaluate our zero-shot VDR approach and baselines on the ViDoRe-v2 benchmark Macé et al. 2025 , an enhanced and more challenging successor to ViDoRe-v1. ViDoRe-v2 comprises nine VDR tasks spanning four languages (English, Spanish, French, and German) and covers a variety of domains, including business, restaurant, and medical. It features real-world, complex queries, for which the best end-to-end supervised VDR model Team 2025 only achieves a moderate nDCG@5 of 0.62. The document

Read the complete text