Hypencoder: Hypernetworks for Information Retrieval

Julian Killingback, Hansi Zeng, Hamed Zamani · 2025

Read Hypencoder: Hypernetworks for Information Retrieval by Julian Killingback, Hansi Zeng, Hamed Zamani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05364.txt.

Introduction

Julian Killingback, Hansi Zeng, and Hamed Zamani propose a new kind of search model in 2025 called the Hypencoder. Most search systems turn a query into a vector and score matches by closeness. That caps how rich a score can be. Their model instead turns each query into a tiny neural network that scores documents itself. A larger network, a hypernetwork, writes that small one.

On standard search tasks the model beat strong dense systems and even rerankers with far more parameters. On hard tasks, like finding a half-remembered item, the lead grew wider. A fast search method lets it scan millions of documents in under sixty milliseconds. The result is both stronger and practical.

Catalog record

CREATORJulian Killingback, Hansi Zeng, Hamed Zamani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05364

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hypencoder: Hypernetworks for Information Retrieval Julian Killingback, Hansi Zeng, Hamed Zamani Abstract . Existing information retrieval systems are largely constrained by their reliance on vector inner products to assess query-document relevance, which naturally limits the expressiveness of the relevance score they can produce. We propose a new paradigm; instead of representing a query as a vector, we use a small neural network that acts as a learned query-specific relevance function. This small neural network takes a document representation as input (in this work we use a single vector) and produces a scalar relevance score. To produce the small neural network we use a hypernetwork, a network that produces the weights of other networks, as our query encoder. We name this category of encoder models Hypencoders. Experiments on in-domain search tasks show that Hypencoders significantly outperform strong dense retrieval models and even surpass reranking models and retrieval models with an order of magnitude more parameters. To assess the extent of Hypencoders’ capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue and instruction-following retrieval tasks. On harder tasks, we find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method, we implement an approximate search algorithm and show that our model is able to retrieve from a corpus of 8.8M documents in under 60 milliseconds. *[inlinelist,1]label=(), Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3729983 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Information retrieval CCS: Computing methodologies Machine learning † † cc-license: by 1. Introduction Efficient neural retrieval models are based on a bi-encoder (or two-tower) architecture, in which queries and documents are represented separately using either high-dimensional sparse ( Zamani et al., 2018 ; Formal et al., 2021 ; Formal et al., 2022 ) or relatively low-dimensional dense vectors ( Karpukhin et al., 2020 ; Xiong et al., 2021 ; Khattab and Zaharia, 2020 ; Zeng et al., 2022 ; Gao and Callan, 2022 ) . These models use simple and lightweight similarity functions, e.g., inner product or cosine similarity, to compute the relevance score for a given pair of query and document representations. We demonstrate theoretically that inner product similarity functions fundamentally limit the types of relevance that retrieval models can express. Specifically, we prove that there is always a set of relevant documents which cannot be perfectly retrieved regardless of the query vector and specific encoder model. Motivated by this theoretical argument, we introduce a new category of retrieval models that can capture complex relationships between query and document representations. Building upon the hypernetwork literature in machine learning ( Ha et al., 2017 ; Sendera et al., 2023 ; von Oswald et al., 2020 ) , we propose Hypencoder–a generic framework that learns a query-dependent multi-layer neural network as a similarity function that is applied to the document representations. In more detail, Hypencoder applies attention-based hypernetwork layers, called hyperhead layers, to the contextualized query embeddings output by a backbone transformer encoder. Each hyperhead layer produces the weight and bias matrices for a neural network layer in the query-dependent similarity network, called the q -net. The q -net is then applied to each document representation, which results in a scalar relevance score. We demonstrate that the Hypencoder framework can be optimized end-to-end and used for efficient retrieval from a large corpus. Specifically, we propose a graph-based greedy search algorithm that approximates exhaustive retrieval using Hypencoder while being substantially more efficient. We conduct extensive experiments on a wide range of datasets to demonstrate the efficacy of Hypencoder. We show that our implementation of Hypencoder outperforms strong single-vector dense and sparse retrieval models on MSMARCO ( Nguyen et al., 2016 ) and TREC Deep Learning Track data ( Craswell et al., 2020b ; Craswell et al., 2020a ) , in addition to complex retrieval tasks, such as TREC DL-Hard ( Mackie et al., 2021 ) , TREC Tip-of-the-Tongue (TOT) Track ( Arguello et al., 2023 ) , and the instruction following dataset FollowIR ( Weller et al., 2024a ) . Across these benchmarks, Hypencoder shows consistent performance gains. Using the proposed approximate retrieval approach on MSMARCO ( Nguyen et al., 2016 ) , with approximately 8.8 million passages, results in an average search latency of 59.6 milliseconds per query on a single NVIDIA L40S GPU. A main advantage of hypernetworks in machine learning is their ability to learn generalizable representations. To demonstrate that Hypencoder also inherits this generalization quality, we evaluated our model under various domain adaptation settings: (1) adaptation to question answering datasets in biomedical and financial domains, and (2) adaptation to other retrieval tasks, including entity and argument retrieval, where Hypencoder again demonstrates superior performance compared to the baselines. We believe that these performance gains are a byproduct of our main contributions: Hypencoder introduces a new way to think about what retrieval and relevance functions can be, it opens a new world of possibilities by bridging the gap between neural networks and retrieval similarity functions. We believe Hypencoder is especially important at this time given the new demands for longer and more complex queries brought on by the widespread usage of large language models and it is our belief that Hypencoder represents an important step towards this goal. To help facilitate this goal, we have open sourced all of our code for training, retrieval, and evaluation. 1 1 1 Available at https://github.com/jfkback/hypencoder-paper encoder pooler d encoder pooler q encoder pooler d encoder pooler q encoder score-head q d s encoder pooler d encoder hyper-head q IP s E_{d} E_{q} s E_{d} input q-net combine E_{d} E_{q} MLP s Vector Similarity Hypencoder Learned Similarity Cross-encoding Similarity Figure 1. Overview comparing Hypencoder to existing retrieval and reranking paradigms. Gray arrows indicate the arrow does not represent an entity, it is the same as what it points to, in contrast black arrows do indicate a unique entity which is always labeled. 2. Related Work Vector Space Models Vector-based models that use sparse vectors have existed for decades, with each index representing a term in the corpus vocabulary. Document-query similarity is computed using measures like \ell^{2} distance, inner product, or cosine similarity, with term weighting methods such as TF-IDF being a substantial focus to improve performance. With the emergence of deep neural networks, focus shifted to learning representations for queries and documents. SNRM by Zamani et al. (2018) was the first deep learning model to retrieve documents from large corpora by learning latent sparse vectors. Following works leveraged pretrained transformer models like BERT ( Devlin et al., 2019 ) using single dense vector representations ( Karpukhin et al., 2020 ) . Recent improvements have focused on training techniques including self-negative mining ( Xiong et al., 2021 ; Qu et al., 2021 ; Prakash et al., 2021 ; Zhan et al., 2021 ; Zeng et al., 2023 ) , data augmentation ( Qu et al., 2021 ; Lin et al., 2023a ) , distillation ( Hofstätter et al., 2020 ; Lin et al., 2023a ; Lin et al., 2020

Read the complete text