NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
Zhuchenyang Liu, Yao Zhang, Yu Xiao · 2026
Read NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval by Zhuchenyang Liu, Yao Zhang, Yu Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12824.txt.
Introduction
Zhuchenyang Liu and co-authors shrink visual document retrieval by splitting how documents and queries are encoded. Big vision-language retrievers use the same huge encoder for both sides, which is slow even for short text queries. Yet documents are visually complex, while queries are plain short strings. Their NanoVDR keeps a frozen 2B teacher to index documents offline. A tiny text-only student encodes queries instead. Simple cosine alignment proved the best distillation aim across many tests.
Catalog record
| CREATOR | Zhuchenyang Liu, Yao Zhang, Yu Xiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52061 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval Zhuchenyang Liu, Yao Zhang, Yu Xiao Abstract Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi 1 1 1 https://huggingface.co/nanovdr/NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32 × \times fewer parameters and 50 × \times lower CPU query latency, at a total training cost under 13 GPU-hours. Figure 1: Motivation and deployment advantage of NanoVDR. (a) Symmetric vs. asymmetric retrieval: Current VDR systems (top) use the same heavy VLM encoder (2B) for both offline document indexing and online query encoding ( > 2,000 ms per query). NanoVDR (bottom) decouples the two: the frozen VLM teacher encodes documents offline, while a distilled text-only student (70M) encodes queries online in \sim 50 ms on CPU. (b) Performance vs. latency: On the ViDoRe benchmark (mean NDCG@5 across v1/v2/v3), NanoVDR models achieve near-teacher accuracy (dashed line) at 50–143 \times lower CPU latency. Bubble size is proportional to model parameter count; the star ( \bigstar ) marks NanoVDR-S-Multi, the multilingual-augmented variant (§ 3.3 ). 1 Introduction Visual document retrieval (VDR) has achieved remarkable effectiveness in retrieving information from visually rich documents—financial reports with charts, scientific papers with figures, industrial manuals with diagrams—by treating each page as an image rather than relying on OCR-based text extraction 4 ; 14 . State-of-the-art systems use Vision-Language Models (VLMs) to encode both queries and document pages into a shared embedding space 4 ; 14 ; 29 ; 17 . However, these systems apply the same heavyweight VLM encoder for both document indexing and query encoding. This results in high computational overhead at query time, requiring multi-billion parameter models and GPU inference even for plain-text queries, and leads to large index storage costs for high-dimensional representations. A key observation is that this design is unnecessarily symmetric: documents are visually complex and genuinely require strong visual understanding, whereas queries are short text strings that carry no visual content. Using a multi-billion parameter VLM to encode text-only queries wastes the model’s visual processing capacity entirely. To exploit this query–document asymmetry, we propose NanoVDR, which decouples the two encoding paths through knowledge distillation (Figure 1 a). A frozen VLM teacher indexes documents offline, producing single-vector visual embeddings; a lightweight text-only student (as small as 69M parameters) encodes queries at inference by mapping them into the teacher’s embedding space via a learned projector. The student requires no vision module and runs on CPU in {\sim} 50 ms, enabling single-vector cosine similarity retrieval. The central design choice is the distillation objective—how to train the student to faithfully represent queries in the teacher’s visual space. Through systematic comparison of six objectives across three backbones and the full 22-dataset ViDoRe benchmark, we make the following contributions: • Asymmetric distillation framework. We propose a framework that distills a frozen 2B VLM teacher into text-only student encoders (69–151M) for VDR. We show that pointwise cosine alignment which directly matches student and teacher query embeddings, outperforms all ranking-based and contrastive objectives. It only pre-cache teacher query embeddings and eliminating corpus-related processing from training entirely. • Extreme efficiency. NanoVDR-S (DistilBERT, 69M) outperforms DSE-Qwen2 (2B) on v2 and v3 with 32 \times fewer parameters and 50 \times lower query latency (Figure 1 b), at a total training cost under 13 GPU-hours. It also keeps the storage efficiency compared to multi-vector methods, which is inherited from the teacher model. • Cross-lingual augmentation. We further identify in asymmetric encoders for VDR task, the cross-lingual transfer is the primary performance bottleneck, rather than cross-modal transfer. We resolve it via query-only multilingual augmentation, raising teacher retention from 92.4% (NanoVDR-S) to 95.1% (NanoVDR-S-Multi) at low cost. Figure 2: Query-centric distillation training of NanoVDR. Left: The frozen VLM teacher pre-caches training query embeddings via text-only inference. Right: The student text encoder is trained to minimize \mathcal{L}_{\text{align}}=1-\cos(\mathbf{v}^{Q}_{t},\mathbf{v}^{Q}_{s}) , requiring no document images or negative sampling. 2 Related Work 2.1 Visual Document Retrieval Visual document retrieval treats document pages as images and uses VLMs for both query and document encoding. ColPali 4 adapts PaliGemma into a ColBERT-style 10 late interaction model, where each document page produces hundreds of token-level embeddings scored via MaxSim. The same work introduced the ViDoRe benchmark spanning diverse document types. DSE 14 takes a single-vector approach, producing one embedding per document screenshot using Qwen2-VL 26 . VisRAG 31 demonstrates retrieval-augmented generation over visual documents. While retrieval quality has steadily improved, each generation of models has grown larger: more recent multi-vector systems based on 4–8B VLMs 29 ; 17 achieve the highest quality but with query latency exceeding 7 seconds on CPU, further widening the efficiency gap. Vision-native encoders such as SigLIP2 24 and Jina-CLIP 11 offer lighter alternatives but substantially lag behind VLM-based approaches on document retrieval tasks. Three concurrent directions attempt to bridge efficiency and visual understanding. VISTA 32 augments a frozen text encoder (BGE-Base, 110M) with a ViT image tokenizer ( {\sim} 196M total), enabling multi-modal retrieval without modifying the text backbone; however, the ViT remains required at inference, and the model has not been evaluated on document-level benchmarks. ModernVBERT 22 builds a purpose-designed 250M vision-language encoder by fusing a SigLIP2 vision encoder with a ModernBERT backbone via early fusion, matching ColPali-level quality at 12 \times fewer parameters; nevertheless, both query and document encoding still require the full vision-language model. SERVAL 16 takes a generate-then-encode approach: a VLM generates textual descriptions of document images, which are then indexed by a standard text encoder. While zero-shot and effective (63.4 NDCG@5 on ViDoRe v2 with a 72B VLM + 7B encoder), the pipeline requires massive VLM inference for every document at indexing time. Our approach differs fundamentally: we distill the VLM’s embedding space directly into a tiny text-only encoder (69M), requiring neither a vision module at inference nor VLM-scale caption generation. 2.2 Knowledge Distillation in Dense Retrieval Knowledge distillation 5 has been extensively applied to dense text retrieval, where using separate encoders for queries and documents is well-established (DPR 9 , ColBERT 10 ). NanoVDR extends this asymmetry across modalities, pairing a VLM document encoder with a text-only query encoder. TAS-B 7 uses topic-aware sampling with balanced training from a cross-encoder teacher. MarginMSE 6 distills pairwise margin scores to train efficient