Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrieval

Hung-Ting Chen, Xiang Liu, Shauli Ravfogel, Eunsol Choi · 2025

Read Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrieval by Hung-Ting Chen, Xiang Liu, Shauli Ravfogel, Eunsol Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02770.txt.

Introduction

Hung-Ting Chen, Xiang Liu, Shauli Ravfogel, and Eunsol Choi question a habit in search systems: turning a query into a single embedding vector. When a query has several good answers pointing in different directions, one vector must average them and can miss most of them. Their 2025 paper first measures how current retrievers suffer as target documents grow more spread out.

Their model, AMER, generates several query vectors one after another, in an autoregressive way, and searches with all of them. On synthetic data it captured multiple target groups perfectly, scoring four times better than a single-vector model. Fine-tuned on real multi-answer data, it gained 4 and 21 percent over baselines on two test sets, with the largest gains where answer embeddings differed most.

Catalog record

CREATORHung-Ting Chen, Xiang Liu, Shauli Ravfogel, Eunsol Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrieval Hung-Ting Chen, Xiang Liu, Shauli Ravfogel, Eunsol Choi Abstract Most text retrievers generate one query vector to retrieve relevant documents. Yet, the conditional distribution of relevant documents for the query may be multimodal, e.g., representing different interpretations of the query. We first quantify the limitations of existing retrievers. All retrievers we evaluate struggle more as the distance between target document embeddings grows. To address this limitation, we develop a new retriever architecture, A utoregressive M ulti- E mbedding R etriever ( AMER ). Our model autoregressively generates multiple query vectors, and all the predicted query vectors are used to retrieve documents from the corpus. We show that on the synthetic vectorized data, the proposed method could capture multiple target distributions perfectly, showing 4x better performance than single embedding model. We also fine-tune our model on real-world multi-answer retrieval datasets and evaluate in-domain. AMER presents 4 and 21% relative gains over single-embedding baselines on two datasets we evaluate on. Furthermore, we consistently observe larger gains on the subset of dataset where the embeddings of the target documents are less similar to each other. We demonstrate the potential of using a multi-query vector retriever and open up a new direction for future work. 1 1 1 Code is avaiable at https://github.com/timchen0618/amer . 1 Introduction As large language models (LLMs) have limited, out-dated parametric knowledge, augmenting knowledge at inference time by prepending retrieved documents has risen as a de facto solution ( Fan et al. 2024 ; Gao et al. 2023 ) . Recovering a diverse set of documents is crucial to provide comprehensive information ( Xu et al. 2023 ) , as an answer providing partial information can be technically correct yet misleading to users. In this work, we study retrieving a diverse set of documents per query. We first analyze the behaviors of existing retrievers ( Izacard et al. 2022 ; Yang et al. 2025b ; Zhang et al. 2025 ; Lee et al. 2025a ) on datasets ( Min et al. 2020 ; Amouyal et al. 2023 ) containing questions that admit multiple valid answers. Such questions arise either as a result of inherent ambiguity, or from cases where the desired answer is a list (e.g. From which country did Seattle Storm make draft selections? ). All retrieval models, trained to maximize the probability of obtaining a single target, show degrading performance as the distance between gold target documents belonging to the same query increases, as shown in Figure 1 . We hypothesize that a single query vector, however well constructed, is insufficient to model multiple target distributions. To address this limitation, we propose a new architecture, A utoregressive M ulti- E mbedding R etriever ( AMER ). Instead of constructing a single vector for each query, our model constructs multiple query vectors per query, autoregressively predicting query vectors one by one. Each predicted query vector is used to search the corpus to obtain a ranked list of documents, which we heuristically aggregate into a single ranked list. This approach directly addresses the limitations of single-query vector retrievers and enables retrieving diverse outputs. We first evaluate our model on a new synthetic dataset, where each input query vector is paired with multiple target vectors. We design transformations (e.g., linear transformations and multi-layer perceptrons (MLPs)) to apply to the input query vector. The transformation yields target vectors that are far away from each other. We show that it is difficult for the single-vector retriever model to capture all target distributions, only retrieving all the target embeddings at most 21% of the time, validating our hypothesis. AMER retrieves all targets perfectly (100% of the time) in this setting, across the diverse vectorized dataset that we created. The results indicate that the proposed multi-vector retriever is more suitable for modeling heterogeneous target distributions. Going beyond the synthetic setting, we further evaluate our model on two real-world text retrieval datasets ( Min et al. 2020 ; Amouyal et al. 2023 ) . In such setting, we expect to see an improvement when the target documents form distinct clusters. We find that AMER exhibits small average performance gains compared to the single-query baseline in these datasets (4%, 21%), much smaller than those observed in the earlier synthetic experiments. However, our method shows more pronounced gains (5%, 144%) in a subset of evaluation data where target gold documents belonging to the same query are farther away from each other in the embedding space. This trend holds true for various base language models (LMs) we tested (e.g. Llama-3 ( Grattafiori et al. 2024 ) , Qwen-3 ( Yang et al. 2025a ) ). Further analysis reveals that target documents belonging to the same query are substantially more similar than distractors in these datasets, surfacing the need for better benchmarks for diversity in retrieval. We will release our code publicly upon publication. 2 Background: Multi-Targets Retrieval Task Given a corpus D and a query q that admits m answers \displaystyle\{a_{1},\dots,a_{m}\} , systems should retrieve a subset D_{q} of D that covers all the m targets. We assume document clusters \displaystyle\{d_{1}\},\dots,\{d_{m}\} , such that \displaystyle\{d_{i}\} covers the answer a_{i} . Most existing dense retrievers employ a bi-encoder architecture where query and document encoders produce their respective embeddings, and document relevance is scored by the similarity between these embeddings. These models are trained using contrastive loss, pushing the query embedding closer to the ground truth document embeddings and away from the negative document embeddings. Therefore, if the answer document clusters are far from each other, it will be difficult for a single query embedding to fit to all of the clusters. Metric Typically retrieval performances are evaluated using Recall @ k , which measures if the top- k retrieved document set D^{k} contains the target document. In datasets where there are multiple distinct valid answers, MRecall @ k ( Min et al. 2021 ) is used, which evaluates the top- k retrieved document set D^{k} against m target documents. • If k\geq m , then MRecall @ k = 1 if D^{k} contains all m answers; else MRecall @ k = 0. • If k<m , then MRecall @ k = 1 if D^{k} contains k answers; else MRecall @ k = 0. Table 1: Data Statistics. We report the number of instances in each data split, and the average number of targets per question in the test set. # Train # Test # Targets AmbigQA 5,044 827 2.58 QAMPARI 32,023 531 6.09 Multi-answer QA Datasets We evaluate on two popular, easy-to-evaluate multi-answer QA datasets, AmbigQA ( Min et al. 2020 ) and QAMPARI ( Amouyal et al. 2023 ) . We discuss other retrieval datasets with multiple targets in Appendix A.1 . AmbigQA dataset contains questions sampled from Natural Questions dataset ( Kwiatkowski et al. 2019 ) that are ambiguous and multiple disambiguated answers of each question. Each question is thus associated with multiple target documents, which answer the question with different disambiguations. 2 2 2 AmbigQA contains some single-answer examples, which we discard. We only keep the multi-answer ones. QAMPARI contains questions with a list of entity answers spanning multiple paragraphs, which necessitates retrieving many documents. We present examples

Read the complete text