LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference

Guangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, Songlin Hu · 2025

Read LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference by Guangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, Songlin Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12260.txt.

Introduction

Search engines built on large language models read documents slowly once, ahead of time, but must handle each fresh query fast. Guangyuan Ma and co-authors attack that bottleneck with LightRetriever. Their design keeps the full large model for encoding documents offline. For the query side, it shrinks the work to little more than a table lookup.

Against a full model running on a high-end graphics card, the lookup approach encodes queries more than a thousand times faster. Whole searches run over ten times quicker end to end. On big standard benchmarks, the light system keeps about ninety-five percent of the heavy system's accuracy. It also holds up across varied search tasks. The authors pitch it as a way to serve model-grade search at ordinary budgets.

Catalog record

CREATORGuangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, Songlin Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference Guangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, Songlin Hu Abstract Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever , a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10× increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance. 1 Introduction Given search queries and candidate documents, text retrieval employs dual-encoder architectures to encode representation vectors and search documents relevant to queries based on query-document similarities ( Salton et al., 1975 ) . The encoded vectors could be condensed into low-dimensional vectors for dense retrieval or sparsified into high-dimensional vectors for sparse retrieval. Dense retrieval ( Karpukhin et al., 2020 ) compresses textual information into dense vectors with parameterized language models (LMs) and retrieves based on inner product similarity. Meanwhile, sparse retrieval ( Robertson et al., 1994 ) characterizes sparse term frequencies on a much broader vocabulary space, and then searches based on lexical overlap and term-based relevance metrics, such as direct TF-based search (Inner product) ( Bai et al., 2020 ; Formal et al., 2021 ) and BM25 ( Robertson et al., 1994 ) . Additionally, hybrid retrieval enhances retrieval abilities by interpolating their similarity scores ( Wang et al., 2021 ) or ranks ( Cormack et al., 2009 ) . Text retrieval is broadly applied in various scenarios, such as web search ( Lu et al., 2022 ) , semantic textual similarity ( Bowman et al., 2015 ) , fact checking ( Thorne et al., 2018 ) , and retrieval augmented generation ( Lewis et al., 2020 ) . Figure 1: LightRetriever targets at extreme query inference speed for LLM-based text retrieval, reducing the workload of query encoding to no more than an embedding lookup. Deploying retrieval systems requires different processing workflows for queries and documents. As shown in Figure 1 , as search candidates, documents are encoded with a document encoder Enc_{d} and then indexed into search databases ( Johnson et al., 2021 ; Robertson et al., 1994 ) . Because documents are obtained before the search operation, they are pre-computable and distributable, which can be indexed offline and distributed anywhere ( Yang et al., 2018a ) . On the contrary, queries are input and encoded in real-time, which are non-pre-computable and non-distributable. The query encoder Enc_{q} needs to be served online. Special GPU accelerators are mandatory if the query encoder is a deeply parameterized LM. Recent state-of-the-art (SOTA) retrievers ( Wang et al., 2024a ; BehnamGhader et al., 2024 ) employ deeply parameterized LLMs as symmetric dual-encoders Enc_{q} and Enc_{d} for enhanced retrieval capabilities. However, serving giant LLMs slows down query inference throughput and depletes online deployment resources ( Kwon et al., 2023 ) . Despite the rapid progress on improving retrieval quality, little prior work has investigated the efficiency challenges of LLM-based retrievers, necessitating special optimizations for query inference efficiency. Modern Transformer-based LMs empower text retrievers with rich contextual modeling through the attention mechanism ( Vaswani et al., 2017 ) , enabling deep interactions across contextual tokens. This capability explains the success of recent LLM-based dual-encoders ( Wang et al., 2024a ; BehnamGhader et al., 2024 ) , but it also means that both queries and documents incur the same computational burden. In contrast, traditional statistical methods such as BM25 ( Robertson et al., 1994 ) rely purely on lexical overlap and token frequency, requiring negligible inference cost while still remaining competitive in many retrieval tasks ( Thakur et al., 2021 ) . This contrast highlights a practical tension: While document encoders can afford heavy pre-computation offline, query encoders must operate under strict online latency and resource constraints. Therefore, it is natural to ask: do queries truly need the same degree of deep contextual modeling as documents? To strike a balance between performances and efficiencies, we propose LightRetriever , a new hybrid retrieval architecture that explicitly breaks the symmetry between document and query encoders. Our key insight is that while documents benefit from the full modeling power of LLMs, queries do not require equally heavy processing to achieve strong retrieval performance. To this end, LightRetriever preserves a full-sized LLM encoder on the document side, but entirely removes the deep modeling on the query side . 1) For dense retrieval, we end-to-end train cacheable token embeddings from the LLM-based query encoder and aggregate them through simple averaging, reducing query inference to a single embedding lookup; 2) For sparse retrieval, we directly map token counts to sparse vectors without any encoding. Dense and sparse scores are linearly interpolated ( Wang et al., 2021 ) for the final hybrid retrieval. LightRetriever achieves retrieval quality mostly comparable to full dual-encoder LLMs, but with orders-of-magnitude faster query inference. Compared to serving a full-sized LLM on an A800 GPU, our method achieves over a thousand times speedup for query encoding, and over 10x total throughput improvements. Importantly, our experiments show that adopting a lightweight query encoder does not mean queries are deprived of deep semantic understanding. Our work tries to transfer the major modeling cost from the query side to the document side, while still preserving the benefits of LLM-powered relevance-based similarity search with query-document interactions. Experiments on large-scale retrieval benchmarks demonstrate that our method generalizes well across diverse LLMs (including Llama-1B, 3B, 8B, and Qwen-1.5B, 3B, 7B) and retrieval benchmarks (including 15 English tasks from BeIR ( Thakur et al., 2021 ) and 8 Chinese tasks from CMTEB Retrieval ( Xiao et al., 2024 ) ), maintaining an average of 95% retrieval performance. 2 Algorithm Figure 2: Contrastive training of full-sized symmetric dense retrieval. A full-sized dual-encoder is used to model both queries and documents. Task-specific instructions are added as common practices ( Su et al., 2023 ; Wang et al., 2024a ) to promote better domain adaptation abilities. 2.1 LM-based text retrieval Definition Given a search query q and candidate documents d=\{d^{+};d^{-}\}\in\mathcal{N}(q) (including irrelevant documents d^{-} and relevant documents d^{+} ) retrieved from indexes \mathcal{N} , LM-based text retrieval aims to learn optimized encoders Enc_{q} and Enc_{d} with parameters \theta , where the similarity of the query vector v_{q}=Enc_{q}(q) and irrelevant document vectors v_{d^{-}}=Enc_{d}(d^{-}) are encouraged to be smaller than those of relevant documents v_{d^{+}}=Enc_{d}(d^{+}) . Assuming the dot product is used as the similarity metric, the above optimization goals can be described as the following objective, \min_{\theta}\max_{d^{-}\in\mathcal{N}(q)}\mathcal{E}(q,d^{+},d^{-};\theta)\text{, where }\mathcal{E}(q,d^{+},d^{-};\theta)=v_{q}\cdot v_{d^{-}}-v_{q}\cdot v_{d^{+}} (1) which lowers the upper boundary of ranking similarity errors \mathcal{E} ( Chen et al., 2009 ) . The above objective can be effectively optimized by contrastive loss ( Cao et al.,

Read the complete text