“Someone Hid It!”: Query-Agnostic Black-Box Attacks on LLM-Based Retrieval Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, Yue Zhao Abstract Large language models (LLMs) have been serving as effective backbones for retrieval systems, including Retrieval-Augmentation-Generation (RAG), Dense Information Retriever (IR), and Agent Memory Retrieval. Recent studies have demonstrated that such LLM-based Retrieval (LLMR) is vulnerable to adversarial attacks, which manipulates documents by token-level injections and enables adversaries to either boost or diminish these documents in retrieval tasks. However, existing attack studies mainly (1) presume a known query is given to the attacker, and (2) highly rely on access to the victim model’s parameters or interactions, which are hardly accessible in real-world scenarios, leading to limited validity. To further explore the secure risks of LLMR, we propose a practical black-box attack method that generates transferable injection tokens based on zero-shot surrogate LLMs without need of victim queries or victim models knowledge. The effectiveness of our attack raises such a robustness issue that similar effects may arise from benign or unintended document edits in the real world. To achieve our attack, we first establish a theoretical framework of LLMR and empirically verify it. Under the framework, we simulate the transferable attack as a min-max problem, and propose an adversarial learning mechanism that finds optimal adversarial tokens with learnable query samples. Our attack is validated to be effective on benchmark datasets across popular LLM retrievers. 1 1 1 Our experimental code is available as the github repository: https://github.com/JetRichardLee/DQA-Learning 1 Introduction Figure 1 : In many practical scenarios, attackers may hope to hide web documents from retrieval systems. These websites usually allow normal public users to edit in format of content contribution or discussion replies. Retrieval system, which aims to efficiently seek most relevant documents for given user queries, not only occupies great importance in applications like search engines and recommendation systems, but also plays a vital role in knowledge techniques like Retrieval-augmented Generation (RAG) ( 36 ) and Information Retrieval (IR) ( 33 ) . With the advance of large-language models (LLMs) ( 31 ; 1 ; 30 ) , retrieval method has extended a new series of strong backbones and stepped into a new stage of high performance, yet also posing potential secure risks in the face of adversarial attacks. These risks could be utilized to adversarially diminish useful information ( 26 ; 28 ) , trigger harmful content by LLM ( 29 ; 38 ) and stealthily manipulate commercial recommendations ( 12 ; 25 ) . In this work we mainly focus on scenarios where the attacker aims to hide a victim document from LLM-based Retrieval (LLMR) , whether in adversarial desires or reasonable intentions (illustrated as Figure 1 ). The above observations have drawn attention to the vulnerability of LLM retrievers, including designs for adaptive adversarial attack methods. However, existing attack methods share two main limitations: (1) Most works focus on white-box settings ( 12 ; 22 ; 25 ) where the attacker has full access to the model’s parameters, which makes such a threat unconvincing in real-world applications. A few studies ( 28 ; 18 ) have explored black-box attack in small NLP models like Bert, yet in lack of effectiveness validated (shown in Section 4 ) for advanced retrieval systems today like LLMR and in need of abundant interactions with the victim system, which is hard to achieve in the real world; (2) In these problem settings, the attacker always has the knowledge of the victim user’s query, and sometimes the document corpus of the victim retriever as well. However, in practical scenarios, it’s nearly impossible for the attacker to have access to the hidden documents corpus, which is held by the retrieval system behind, or to perturb the document immediately after receiving the explicit user’s query at test-time. These limitations in practicability largely reduce existing works’ impact on understanding the vulnerability of LLMR, and further drive us to raise the question: " How vulnerable are LLM retrievers in the face of black-box adversarial attacks in practical scenarios ?" To answer this question and address limitations of existing studies, we propose a complete black-box attack method, which learns transferable tokens on a surrogate LLM with only knowledge of victim documents, and particularly focuses on absenting victim information sources. To be specific, (1) We first establish a theoretical framework based on empirical observations, and analyze the three factors for achieving black-box transferable attack: derivation of victim document embedding, precision of surrogate LLM and precision of the target LLM; (2) To yield sufficient topic derivation, our attack samples potential user queries utilizing casual LLMs, and performs an adversarial learning between the sampled query and the victim documents on the surrogate model. (3) To further strengthen optimality, we adapts two technical tricks: imbalance on query population and word-embedding as surrogates. Compared with existing attacks, our method have three main advantages: (1) Practical attacker capacity settings , which simulate a potential attacker that can only read and slightly edit the knowledge documents as a normal user (e.g., Wikipedia contributor, reddit thread commenter) without any knowledge of retriever models, builders or users; (2)Theoretical insights on attack transferability , which is methodological deducted based on our established framework of topic embedding clusters of LLMs and formulated into necessary conditions for achieving transferable attacks on surrogate models; (3) Zero-shot transferable attack over different LLM retrievers , which doesn’t require high interactions with a specific victim model like other black-box attacks, and once learnt, could be destructive against various LLM retrievers globally. In other words, once a document is polluted by our injected tokens, all retrievers would be possibly influenced to retrieve it at the same time. This paper’s contributions are summarized as follows: 1. We first investigate the vulnerability of LLMR in the face of query-agnostic black-box settings , where the attacker has no access to the victim model, the document corpus, or user queries in any kind of format. 2. We establish a theoretical framework on LLM retrievers with empirical verification, and propose an adversarial learning method between victim documents and sampled query candidates on a surrogate model with zero-shot transferability . 3. Our method is validated on benchmark datasets to be globally effective across various victim LLM retrievers with the same injected tokens for a specific document, and outperforms other existing attack baselines. 2 Preliminaries and Problem Formulation Figure 2 : Illustration of LLM-based Retrieval. Documents in the corpus are firstly embedded in the last-hidden embeddings and stored. When a user query comes and get embedded, it matches relevant documents in embedding similarity in high efficiency. LLM as Retrievers Nowadays advancing large language models are mostly built on transformer architectures. For simplification of notation, in this paper we skip the setting of tokenizers, and represent all context datas (document or query) as a series of tokens. When a context X\in\mathbb{R}^{L} is required for retrieval embedding on LLM f , it first forward into a word-embedding layer f_{0}:\mathbb{R}\rightarrow\mathbb{R}^{m_{0}} : \textbf{X}_{0}=f_{0}(X) The processed \textbf{X}_{0}\in\mathbb{R}^{L\times m_{0}} is called word-embeddings (later used in Section 3.3 ). \mathbf{X}_{0} afterwards is forward into T transformer blocks f_{[T]}=\{f_{1},\dots,f_{T}\} : \mathbf{X}_{i}=f_{i}(\mathbf{X}_{i-1}),\forall i\in[1,T] \mathbf{X}_{[T]}\in\mathbb{R}^{L\times m} are described as the hidden in each block. When applying retrieval tasks, the last hidden \mathbf{X}_{T} is adopted for the retrieval embedding through a last-hidden-pool function: f(X)=\text{hidden-pool}(\mathbf{X}_{T}) Depending on different settings, the pool function is usually a weighted sum in final attention or simply the last token hidden \mathbf{X}_{T,L} . When evaluate the relevance of two context X_{i} and X_{j} , a statistic similarity function sim (usually cosine-similarity) is applied to f(X_{i}) and f(X_{j}) . This statistic evaluation allows the retrieval system to preprocess the embedding of the documents (usually in large amount), and only forward LLM once for the query when a user inquiries. This ensures the efficiency of the LLMR system when holding millions of documents prepared for retrieval. Token Injection Attack Token injection attack is one of the most popular approaches in LLM adversarial attack. For a given context d , the attacker injects a few attack tokens at the end of d to achieve the LLM’s performance decrease or unsafe behaviors like jailbreaks. Here we mainly introduce Greedy Coordinate Gradient (GCG), a white-box optimizable injection attack, for our information retrieval attack purpose. Specifically, the document d is firstly injected with m initial tokens (such as multiple * ) as the initial d^{\prime} : d^{\prime}=d\oplus t_{[m]},\{t_{i}=\text{``*"}\} Then the attacker loops greedy learning steps to optimize the injected tokens regarding a loss \mathcal{L} depends on a specific task performed on d^{\prime} . At each greedy step, \mathcal{L}(g(d^{\prime})) is backward to obtain the each token’s gradients at the word embedding layer, which correspond to a specific gradient for every potential word w (or token) in every t_{i} ’s position: \partial w^{i}=\frac{\partial\mathcal{L}(g(d^{\prime}))}{\partial t_{i}}|_{t_{i}=w},\forall w\in\mathcal{W} \mathcal{W} defines the whole vocabulary set that the victim LLM supports. Then the attacker pick up k words w_{[k]}^{i} that occupy top- k greatest gradients \partial w^{i} for every injected token t_{i} , and randomly choose one to replace the original one. Such a replacement constructs a greedily optimized sample \{w_{i}\} . The attacker makes several such samples, and eventually picks the one with lowest loss as the new d^{\prime} . t_{i}^{\prime}\leftarrow w_{i}\in\arg\min_{\{w_{i}\}}\mathcal{L}(g(d\oplus\{w_{i}\})) Note although GCG requires gradient backpropagation, the optimization is still a heuristic search in discrete forms. Problem Formulation Given a LLM model serving as a embedding function f , for a document d belongs to a document corrupt D , and a query document, the attacker manipulates the document to a perturbed version d^{\prime} , which could be hard for the model f to pick as top- k from D^{\prime}=D\cup\{d^{\prime}\}\setminus\{d\} in a embedding similarity order. We emphasize that in a complete black-box scenario, the capacity of attacker could be relatively limited: 1. The query q is unknown to the attacker, since it’s performed as a post-attack action by unknown victim user; 2. The LLM retriever model f is a complete black-box, which means the attacker has no knowledge of f including architecture and parameters, nor access to query it; 3. The document corpus D is hidden to the attacker. The candidate documents are previously collected and hold by the retrieval system in an unknown manner; 4. The injected tokens applied to d is limited in amount, which could be expressed as a similarity constraint of d and d^{\prime} with threshold \delta . With the above settings, we formulate our attack problem as followed: \displaystyle\arg\min_{d^{\prime}}\mathbb{I}(d^{\prime}\in\text{top}_{k}(\text{sim}(f(q),f(D^{\prime})))) (1) \displaystyle s.t.,D^{\prime}=D\cup\{d^{\prime}\}\setminus\{d\},\text{dis}(d,d^{\prime})\leq\delta 3 Document-Query Adversarial Learning Figure 3 : We sample 40 knowledge contexts on each of four roughly-defined topics and visualize their LLMR embeddings by Principal Component Analysis (PCA) reduction. Embeddings of contexts within the same topic ( R :science, G :politic, Y :movie, B :architecture) tend to cluster together in the embedding space. The main difficulty we faced in the stated problem is the black-box setting of victim LLM retriever, which gives neither parameters nor abundant query access. Therefore, we look for ways to learn our injection attack on on a surrogate LLM retriever. However, due to their high quantities in parameters, it is hard to build transferability between LLM models without enough insights. This hard-point draws us to do prior experiments to study the pattern of LLM embeddings, where we find a interesting yet rational phenomenon called topic cluster : the LLM-generated embeddings of context datas in the same topics tend to cluster, as shown in Figure 3 . Based on this observation, (1) we first raise a theoretical definition called " \epsilon-p_{\epsilon} -Precise" retriever to quantify how LLM assembles the embeddings in same topics for the retrieval task, and verify this definition by simulation experiments. (2) Then utilizing this defined property, we are able to make methodology deduction on the transferability of the surrogate attack, and equalize it into a min-max problem. (3) Aiming to solve the min-max problem, we therefore design our Document-Query Adversarial Learning(DQ-A Learning) by optimizing both victim documents and sampled queries in an adversarial learning way. (4) Finally, we empirically find that by adopting the word-embedding for attack instead of last-hidden on LLM, the transferable attack works more effectively, which helps to strengthen our method. 3.1 Definition of \epsilon - p_{\epsilon} -Precise Retriever Topic Name p_{0} p_{0.1} p_{0.2} p_{0.3} In Sim. Out Sim. Engin. Chemical 100% 99.6% 91.7% 87.8% [0.67,0.90] [-0.07,0.58] Computer 92.6% 87.0% 87.0% 66.5% [0.49,0.87] [-0.05,0.63] Electric 92.6% 89.6% 83.0% 47.8% [0.50,0.89] [-0.05,0.63] Biology 99.1% 93.9% 88.3% 86.1% [0.55,0.85] [-0.09,0.56] Movies Action 89.1% 77.0% 29.1% 0.4% [0.37,0.87] [-0.06,0.60] Horror 100% 94.3% 90.9% 82.2% [0.59,0.88] [-0.04,0.59] Romantic 88.3% 87.4% 79.6% 28.7% [0.44,0.88] [-0.05,0.63] Comedy 87.0% 72.6% 33.9% 0.0% [0.32,0.82] [-0.07,0.63] Achite. Baroque 99.6% 94.3% 91.3% 87.4% [0.70,0.94] [-0.03,0.72] Gothic 100% 96.1% 91.3% 87.0% [0.71,0.91] [-0.05,0.71] Renaissance 100% 90.9% 87.0% 87.0% [0.64,0.89] [-0.02,0.72] Modern 100% 100% 87.8% 86.5% [0.65,0.88] [-0.03,0.54] Geogra. American 84.8% 78.7% 50.4% 5.2% [0.40,0.86] [-0.07,0.63] European 92.2% 87.0% 86.5% 78.3% [0.57,0.89] [-0.04,0.69] Asian 89.1% 87.0% 87.0% 80.0% [0.54,0.85] [-0.06,0.68] African 91.3% 87.4% 87.0% 83.0% [0.58,0.85] [-0.09,0.69] Sports Soccer 100% 97.8% 97.8% 91.7% [0.71,0.89] [-0.09,0.64] Baseball 100% 93.0% 91.7% 91.7% [0.66,0.92] [-0.05,0.64] Golf 90.9% 90.0% 68.3% 15.7% [0.44,0.91] [-0.08,0.63] Basketball 87.8% 82.6% 13.9% 0.0% [0.31,0.59] [-0.06,0.61] Politics American 22.2% 0.0% 0.0% 0.0% [0.44,0.81] [-0.04,0.62] Japanese 92.2% 88.3% 80.4% 36.5% [0.44,0.81] [-0.04,0.62] Chinese 1.3% 0.0% 0.0% 0.0% [0.05,0.77] [-0.06,0.55] European 76.5% 30.9% 1.7% 0.0% [0.28,0.87] [-0.06,0.63] Table 1 : 24 context topics’ p_{\epsilon} on Qwen1.5-7B-Ins. retriever. "In Sim" describes the minimum and maximum of similarity between every pair within the topic corpus (10 \times 10), and "Out Sim" for every pair of one within the topic and one in other 23 topics (10 \times 230). Results on other LLMs are reported in Appendix. Under the complete black-box setting, we don’t know the specific inner properties of f , so we instead restrict its behavior as a well-performed retriever by assuming it always generates similar embeddings for contexts in the same topic while dissimilar ones for others: Definition 3.1 ( \epsilon - p_{\epsilon} -Precise Retriever) . Given an global text set \mathbb{X} , a subset \mathcal{X}\subseteq\mathbb{X} which contain contexts \{X_{1},X_{2},...\} within a same topic distribution, not limiting to query or document, and an complementary context set \mathcal{X}^{\prime}=\mathbb{X}-\mathcal{X} which consists of remaining contexts not in same topics with \mathcal{X} , we define a retriever to be \epsilon - p_{\epsilon} -Precise over \mathcal{X} if there are at least p_{\epsilon} fraction of \mathcal{X}^{\prime} (note as \mathcal{X}^{\prime}_{\epsilon} ) satisfying: \displaystyle\text{sim}(f(X_{i}),f(X_{j}))\geq\text{sim} \displaystyle(f(X_{k}),f(X^{\prime}))+\epsilon, (2) \displaystyle\forall X_{i,j,k}\in\mathcal{X}, \displaystyle X^{\prime}\in\mathcal{X}_{\epsilon} The number \epsilon\in[0,1] describes a similarity gap between documents within and without the topics in probability of p_{\epsilon} . To verify the rationality of this statement, we conduct an experiment: we utilize a Casual LLM to generate 240 knowledge documents in 24 different topics (10 for each), and collect their embeddings on a LLM retriever. Then we test each topic’s p_{0} , p_{0.1} , p_{0.2} and p_{0.3} against other 23 topics as Definition 3.1 described. The result is reported in Table 1 , where most of the topics exhibit in a tight embedding cluster with considerable p_{\epsilon} . This result effectively supports our assumption to be truthful. Under this definition, we assume the victim model f is \epsilon_{f} - p_{\epsilon}^{f} -Precise over the topic \mathcal{X} containing d and q , and we have access to a surrogate model g that is \epsilon_{g} - p_{\epsilon}^{g} -Precise over \mathcal{X} . With these assumptions, we then methodologically solve our attack problem. 3.2 Methodology for Adversarial Learning Figure 4 : The DQ-A learning pipeline of our attack method. Query samples are first generated by a third party casual LLM. Then in every learning step, injected document tokens are first optimized away from queries, and all queries tokens are optimized towards the document. Both surrogate and Casual LLMs require no learning. First noticing the D is unknown to us, and we also have no access to other document in D . Therefore, we transfer the attack target in Eq. 1 to minimize the similarity between d^{\prime} and q : \arg\min_{d^{\prime}}\text{sim}(f(d^{\prime}),f(q)) (3) Since we don’t know the specific ground truth q that the victim user uses, we instead surrogate Eq. 3 by an upper bound: \displaystyle\max_{X\in\mathcal{X}}\text{sim}(f(d^{\prime}),f(X))\geq\text{sim}(f(d^{\prime}),f(q)) (4) Therefore, we seek to optimize \max_{X\in\mathcal{X}}\text{sim}(f(d^{\prime}),f(X)) instead. Despite serving as an approximate bound of target query, we further analysis how this max term benefits the transferable of attack when we conduct on the surrogate g . Lemma 3.2 (Transferability of Attack on g to f ) . After adversarial optimization, if \max_{x\in\mathcal{X}}\text{sim}(g(d^{\prime}),g(X)) is optimized lower than \min_{X_{i},X_{j}\in\mathcal{X}}\text{sim}(g(X_{i}),g(X_{j}))-\epsilon_{g} , and then d^{\prime} could be identified as d^{\prime}\in\mathcal{X}^{\prime} , which leads to a decrease on embeddings similarity of f with probability p_{\epsilon}^{g} . We summarize as: \displaystyle\underbrace{\max_{x\in\mathcal{X}}\text{sim}(g(d^{\prime}),g(X)}_{\text{Optimization Objective}} \displaystyle\leq\underbrace{\min_{X_{i},X_{j}\in\mathcal{X}}\text{sim}(g(X_{i}),g(X_{j}))-\epsilon_{g}}_{\text{Lower Bound for Embeddings Within $q$'s Topic}} (5) \displaystyle\overset{p_{\epsilon}^{f}}{\longrightarrow}\underbrace{\text{sim}(f(d^{\prime}),f(q))}_{\text{Transferred Attack Effect}} \displaystyle\leq\underbrace{\min_{X_{i},X_{j}\in\mathcal{X}}\text{sim}(g(X_{i}),g(X_{j}))-\epsilon_{f}}_{\text{Upper Bound for Embeddings Outside $q$'s Topic}} The above lemma indicates that as f is better as the retrieval embedder, i.e., a larger \epsilon_{f} with high p_{\epsilon}^{f} exists, the transferred attack is more likely to be effective; and as g is better as the embedder, i.e., a nonzero \epsilon_{f} with high p_{\epsilon}^{f} exists, the transferred attack is easier to achieve. This maximum objective term aligns with the upper bound in Eq. 4 . Therefore we set it as our attack’s optimization objective to replace Eq. 3 : \arg\min_{d^{\prime}}\max_{X\in\mathcal{X}}\text{sim}(g(d^{\prime}),g(X)) (6) In practice, we cannot get all possible contexts within the topic \mathcal{X} to calculate the maximum. Therefore, recalling the min-max problem solution raised in Generative Adversarial Nets (GAN) ( 6 ) , we instead approximate Eq. 6 into two recursive equations: \displaystyle X \displaystyle=\arg\max_{X\in\mathcal{X}}\text{sim}(g(d^{\prime}),g(X)) (7) \displaystyle d^{\prime} \displaystyle=\arg\min_{d^{\prime}}\text{sim}(g(d^{\prime}),g(X)) To solve this pair of equations, we could first sample an initial query X that is relevant to d through LLM (optional to be g ) generation, and adopt the classic adversarial learning to optimize the document and the query with their losses opposite on the similarity. However, we need to note that Eq. 7 is not equivalent to Eq. 6 , but instead a necessary (or "weaker") condition to Eq. 6 . Intuitively, it just describes a first-order stationary point, and only when the g function is convex on \mathcal{X} , it equals to the global min-max point of Eq. 6 , which is indeed hardly to happen in LLMR cases. To better adapt our method in such nonconvex cases, we take reference to Two-Time-Scale GDA ( 14 ) , which proves that it is helpful to set inner minimization steps larger than the outer maximization steps. This claim suggests us to apply different learning rate \eta_{d} and \eta_{X} as step sizes to the d^{\prime} and X , specifically with \eta_{X}>\eta_{d} . However in the GCG algorithm which is adopted to optimize d and X , there exists two problems: (1) Each step of GCG learning is a heuristic search and no actual learning rate could be applied, which we have stated in Section 2 ; (2) The optimizable length of X should be pre-set to achieve efficient GCG learning, however, would naturally limit the learning effect of X and make its embedding hard to deviate much in the designed learning steps. Therefore, we choose to turn this step-scale difference into a population difference: instead of learning one sample X , we make N diverse samplings: \mathcal{S}=\{S_{i}=h(d\oplus p_{i}),i\in[1,N]\} (8) where p_{[N]} are crafted prompts to diversely generate related queries (described in Appendix), and h is a third generative LLM. By optimizing each sampled query in \mathcal{S} in fix steps and optimizing X with the maximum of query’s loss, our Document-Queries Adversarial (DQ-A) learning turns Eq. 7 into the adversarial losses: \displaystyle\mathcal{L}_{d} \displaystyle=\max_{S\in\mathcal{S}}\text{sim}(g(d^{\prime}),g(S)) (9) \displaystyle\mathcal{L}_{\text{S}} \displaystyle=-\sum_{S\in\mathcal{S}}\text{sim}(g(d^{\prime}),g(S)) In GCG learning, we optimize every token in the sampled queries for the loss \mathcal{L}_{S} , considering they don’t need to be in practice use. While for the victim document, to simulate a normal user’s edition, we follow the original GCG injection process that only optimize fixed injected tokens on \mathcal{L}_{d} throughout the adversarial learning. Intuitively, this difference in tokens amount may also help for simulating \eta_{X}>\eta_{d} . Afterwards, the learnt adversarial tokens of d^{\prime} are the final injection attack tokens we select. 3.3 Surrogate Hidden with Word-Embedding During the experiments we also empirically find an interesting phenomenon: if we use word-embedding (the first hidden by word-embedding layer) of the input document and the sampled queries to perform the adversarial learning as surrogate of the last hidden, it will offer an even more effective and transferable attack on the target black-box model g ’s performance by retrieving with g^{\prime}s last hidden. We analyze this may due to the abundant parameters of the LLM, and when projected back to the input context, the distance optimized in the latter layer is less influential than the distance optimized in the shallow layer, which decrease the former attack’s effectiveness and transferability. Therefore, we use the word-embedding of surrogate model f instead of its last layer hidden to execute the steps in Section 3.2 : \displaystyle\tilde{g}(X) \displaystyle=\mathbf{X}_{0}^{g}\in\mathbb{R}^{L\times m_{0}^{g}} (10) \displaystyle\tilde{\text{sim}}(\tilde{g}(X),\tilde{g}(X^{\prime})) \displaystyle=\frac{\sum\text{sim}(\mathbf{X}^{g}_{0,i},\mathbf{X}^{\prime g}_{0,j})}{LL^{\prime}} \tilde{g} and \tilde{\text{sim}} stand for the word-embedding-surrogate version of embedding function and similarity matching metric. In the ablation study we will also compare the performance of DQ-A on word-embedding and on last-hidden. 4 Experiments 4.1 Experiment Setting In our experiments, we use four datasets: "economics", "psychology", "biology" and "robotics", from a popular and challenging benchmark BRIGHT ( 10 ) to validate our attack’s influence on retrievers based on popular LLM models, including: 1. Classic opensource base models suggested by BRIGHT: Qwen-1.5-7B, SFR-Embedding-Mistral and E5-Mistral-7B-Instruct; 2. High-download embedding LLM on huggingface: Embedding-Gemma-300M, Jina-Embeddings-v3, Granite-Embedding-r2 and Qwen3-Embedding-0.6B. In ablation study, we test on three best-performance models: jinaai, gemma and Qwen1.5. We use the Qwen-1.5-4B-Casual both as our base surrogate model for learning attacks and casual model for generation. We adopt the word-embedding surrogate (Sec. 3.3 ) as default, and later analysis its impact in ablation study. For the baselines, we choose three typical attack methods: (1) Greedy Coordinate Gradient (GCG) ( 12 ) on the same surrogate model to transfer attack, (2) PRADA ( 28 ) on the same surrogate model, and (3) Poisoned-RAG ( 38 ) , who is prompted (described in Appendix) to generate tokens that diminish the document. Due to designed nature, we provide the ground-truth query for the three baselines. Additionally, we provide the victim model’s query access and 10% of the queries/documents dataset for PRADA to do alignment training, which is achieved in LoRa finetuning process. Note while additional information creates a more beneficial setting for these baselines, the superiority of our method will be validated by following experiments even in face of such imbalanced comparison. We adopt 4 metrics for evaluation: NDCG@25, NDCG@50, Recall@25 and Recall@50. Recall measures if the ground-truth document is retrieved into the top 25/50, while NDCG suggests a rank-relevant metric. We apply both baselines our attacks on all ground-truth documents and evaluated how queries’ retrieval performance drops compared with original results (lower is better \downarrow ). We set the injected token amounts to 10 in our general experiment. For results not lower than 0.5%, we mark them in gray, indicating insufficient attacks. 4.2 Experiments Results General Performance The performance of LLM retrievers on original documents and attacked documents in the default setting is shown in Table 2 . In general, our attack achieves the highest performance decrease compared with other attack methods. In 6 of 7 retrievers, our black-box attack can lead to an obvious decrease in all four averaged metrics, especially around a 2%-10% drop in Recall@25 and a 1%-7% drop in Recall@50. In dataset Robotics, our attack can achieve even an 8% performance drop for Qwen1.5 and 6% for Jinaai in Recall@50, which reduces these best performing retrievers’ around 30% and 20% drop in fraction of their original performance. Only on Qwen3-Emb-0.6B, both our attack and other baselines find it hard to achieve a significant attack effect. We analyze that Qwen3-Emb is potentially post-trained against document injection attack for robustness, while this may also contribute to its lower performance compared with other small-scale LLMs like Jinaai and Emb-Gemma. This observed trade-off between robustness and precision further conforms with our Lemma 3.2 in Section 3.2 . In conclusion, considering our strict limitation in injected tokens (suffix only) and no post-picking or crafting in attack samples, these decreasing effects constitute a strong evidence that LLMR could be vulnerable to adversarial attack even in complete black-box scenarios. We also notice that while Posioned-RAG behaves mostly bad, its oppositely helps to boosts the victim document even prompted to diminish it. One potential reason is that LLM generation task is mostly based on relevance, which may generate related tokens even required not to do so. Victim Models Attacks Economic Psychology Biology Robotics Average NDCG Recall NDCG Recall NDCG Recall NDCG Recall NDCG Recall @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 @25 @50 QWen1.5-7B-Ins Original 0.212 0.237 0.314 0.409 0.272 0.292 0.405 0.493 0.336 0.359 0.469 0.553 0.158 0.177 0.245 0.311 0.245 0.266 0.358 0.442 GCG* 0.184 0.209 0.291 0.393 0.259 0.281 0.398 0.493 0.330 0.355 0.468 0.552 0.138 0.152 0.212 0.259 0.228 0.249 0.342 0.424 PRADA* 0.189 0.215 0.261 0.368 0.263 0.291 0.390 0.501 0.331 0.357 0.469 0.564 0.134 0.155 0.194 0.268 0.229 0.255 0.329 0.425 Poi.-RAG* 0.233 0.260 0.340 0.454 0.308 0.327 0.446 0.527 0.357 0.385 0.493 0.596 0.229 0.251 0.309 0.399 0.282 0.306 0.397 0.494 DQ-A(Ours) 0.132 0.153 0.223 0.309 0.222 0.245 0.347 0.440 0.281 0.311 0.417 0.519 0.108 0.126 0.152 0.231 0.186 0.209 0.285 0.375 SF-Mis.-7B Original 0.203 0.224 0.311 0.395 0.235 0.261 0.418 0.519 0.238 0.270 0.348 0.476 0.187 0.198 0.289 0.321 0.216 0.238 0.342 0.428 GCG* 0.199 0.220 0.303 0.396 0.239 0.266 0.441 0.542 0.227 0.258 0.338 0.453 0.159 0.177 0.245 0.312 0.206 0.230 0.332 0.426 PRADA* 0.199 0.219 0.315 0.403 0.235 0.260 0.431 0.526 0.217 0.256 0.317 0.459 0.161 0.176 0.249 0.309 0.203 0.228 0.328 0.424 Poi.-RAG* 0.242 0.263 0.346 0.434 0.269 0.289 0.455 0.529 0.249 0.280 0.365 0.467 0.272 0.290 0.360 0.418 0.258 0.281 0.382 0.462 DQ-A(Ours) 0.188 0.210 0.297 0.392 0.213 0.243 0.384 0.495 0.205 0.234 0.310 0.417 0.149 0.171 0.262 0.314 0.189 0.215 0.313 0.405 e5-Mis.-7B Original 0.182 0.198 0.294 0.365 0.199 0.229 0.346 0.457 0.225 0.259 0.326 0.451 0.183 0.197 0.269 0.311 0.197 0.221 0.309 0.396 GCG* 0.158 0.183 0.252 0.355 0.190 0.224 0.324 0.462 0.212 0.242 0.311 0.417 0.137 0.159 0.216 0.301 0.174 0.202 0.276 0.384 PRADA* 0.166 0.186 0.266 0.348 0.184 0.213 0.323 0.431 0.203 0.231 0.300 0.401 0.142 0.166 0.217 0.302 0.174 0.199 0.277 0.371 Poi.-RAG* 0.197 0.220 0.305 0.404 0.225 0.246 0.377 0.465 0.229 0.266 0.324 0.461 0.272 0.288 0.366 0.419 0.231 0.255 0.343 0.437 DQ-A(Ours) 0.145 0.173 0.237 0.349 0.161 0.194 0.280 0.410 0.188 0.211 0.287 0.363 0.125 0.145 0.224 0.295 0.155 0.181 0.257 0.354 Emb.-Gemma Original 0.225 0.245 0.307 0.395 0.252 0.285 0.395 0.523 0.223 0.254 0.313 0.420 0.178 0.197 0.262 0.336 0.220 0.245 0.319 0.419 GCG* 0.203 0.222 0.293 0.375 0.234 0.267 0.360 0.488 0.213 0.244 0.304 0.414 0.156 0.180 0.240 0.330 0.202 0.228 0.299 0.402 PRADA* 0.207 0.229 0.285 0.381 0.243 0.265 0.375 0.487 0.213 0.242 0.307 0.416 0.157 0.181 0.232 0.324 0.205 0.229 0.300 0.402 Poi.-RAG* 0.240 0.272 0.308 0.441 0.278 0.312 0.456 0.561 0.231 0.262 0.330 0.440 0.217 0.237 0.322 0.393 0.242 0.271 0.354 0.459 DQ-A(Ours) 0.191 0.214 0.278 0.374 0.229 0.259 0.363 0.486 0.209 0.238 0.300 0.402 0.160 0.183 0.251 0.334 0.198 0.224 0.298 0.399 jina-Emb.-v3 Original 0.225 0.248 0.360 0.453 0.274 0.295 0.422 0.489 0.222 0.249 0.325 0.422 0.176 0.200 0.275 0.371 0.224 0.248 0.346 0.434 GCG* 0.223 0.246 0.370 0.459 0.274 0.295 0.422 0.482 0.213 0.237 0.317 0.399 0.162 0.184 0.261 0.346 0.218 0.241 0.343 0.422 PRADA* 0.225 0.245 0.373 0.450 0.273 0.299 0.413 0.497 0.211 0.236 0.316 0.404 0.169 0.184 0.268 0.329 0.220 0.241 0.343 0.420 Poi-RAG* 0.241 0.264 0.383 0.473 0.294 0.320 0.406 0.503 0.214 0.245 0.317 0.427 0.235 0.261 0.328 0.425 0.246 0.273 0.356 0.457 DQ-A(Ours) 0.221 0.243 0.351 0.437 0.264 0.288 0.410 0.487 0.214 0.239 0.303 0.391 0.152 0.168 0.253 0.318 0.213 0.235 0.329 0.408 Granite-Emb.-R2 Original 0.187 0.210 0.241 0.342 0.182 0.203 0.312 0.396 0.158 0.181 0.242 0.325 0.128 0.146 0.200 0.272 0.164 0.185 0.249 0.334 GCG* 0.174 0.198 0.233 0.335 0.182 0.203 0.325 0.398 0.189 0.220 0.298 0.402 0.114 0.132 0.203 0.270 0.165 0.188 0.265 0.351 PRADA* 0.182 0.200 0.255 0.338 0.180 0.201 0.318 0.392 0.159 0.183 0.239 0.326 0.108 0.132 0.175 0.257 0.157 0.179 0.247 0.328 Poi-RAG* 0.196 0.221 0.256 0.364 0.188 0.214 0.316 0.409 0.159 0.186 0.236 0.330 0.162 0.184 0.234 0.321 0.176 0.201 0.261 0.356 DQ-A(Ours) 0.157 0.179 0.231 0.326 0.177 0.197 0.312 0.386 0.152 0.179 0.229 0.326 0.111 0.130 0.183 0.254 0.149 0.171 0.239 0.323 Qwen3-Emb.-0.6B Original 0.198 0.211 0.293 0.344 0.197 0.218 0.343 0.419 0.159 0.177 0.242 0.305 0.133 0.149 0.203 0.264 0.172 0.189 0.270 0.333 GCG* 0.194 0.208 0.288 0.342 0.203 0.223 0.341 0.412 0.152 0.168 0.215 0.296 0.135 0.150 0.204 0.259 0.171 0.187 0.262 0.327 PRADA* 0.192 0.205 0.288 0.350 0.203 0.224 0.340 0.413 0.149 0.169 0.219 0.292 0.136 0.148 0.216 0.258 0.170 0.187 0.266 0.328 Poi-RAG* 0.204 0.222 0.294 0.363 0.207 0.341 0.340 0.447 0.156 0.176 0.234 0.305 0.175 0.193 0.269 0.333 0.186 0.233 0.284 0.362 DQ-A(Ours) 0.192 0.212 0.288 0.359 0.209 0.230 0.341 0.417 0.146 0.165 0.226 0.292 0.128 0.142 0.200 0.246 0.169 0.187 0.264 0.329 Table 2 : Performance drops after attack( \downarrow ). The best results among baselines are marked in bold and results are considered with no effect are marked in gray . The last four columns are the average results of the former four datasets for general comparison. "*" means this method requires the ground-truth query for the victim document. (a) (b) (c) Figure 5 : Impact of Different |\mathcal{S}| Ablation Study of DQ-A Learning To validate and study the effectiveness of our proposed adversarial learning framework and the word-embedding-surrogate strategy, we further compare our method with two ablation variates: DQ-A-H which doesn’t apply word-embedding-surrogate strategy, and S-GCG which further doesn’t apply adversarial learning but only GCG with sampled initial queries and summed similarities as the document loss. The test results are exhibited in Table 3 , where we can find that DQ-A-H’s attack generally decreases more performance than S-GCG, which validates the effectiveness of our adversarial learning setting. We also notice in a few columns DQ-A-H behaves a little worse, and analyze this may because the learned queries are not properly restricted to the original topic. Therefore while their embeddings chase to the polluted document and no longer lay in original topic cluster, their roles as topic-rejecters become less effective. Comparing the DQ-A and DQ-A-H, we find both baselines have some better attack results, while DQ-A slightly outperforms. This comparison indicates that while the word-embedding strategy has effects as Section 3.3 states, most of the effectiveness still comes from our adversarial learning design. Attacks Economics Psychology Biology Robotics R@25 R@50 R@25 R@50 R@25 R@50 R@25 R@50 jina S-GCG 0.358 0.448 0.414 0.484 0.312 0.422 0.272 0.328 DQ-A-H. 0.360 0.444 0.412 0.484 0.308 0.426 0.245 0.323 DQ-A 0.351 0.437 0.410 0.487 0.303 0.391 0.253 0.318 gemma S-GCG 0.297 0.374 0.396 0.496 0.302 0.423 0.268 0.342 DQ-A-H. 0.282 0.360 0.378 0.495 0.308 0.408 0.238 0.344 DQ-A 0.278 0.374 0.363 0.486 0.300 0.402 0.251 0.334 Qwen S-GCG 0.299 0.376 0.407 0.520 0.478 0.588 0.181 0.284 DQ-A-H 0.289 0.401 0.382 0.511 0.482 0.537 0.214 0.286 DQ-A 0.223 0.309 0.347 0.440 0.417 0.519 0.152 0.231 Table 3 : Performance for Ablation Baselines( \downarrow ) Impact of Sampling Amount To study the impact of the amount of sample queries in our DQ-A learning and validate the population effect we raised in Section 3.2 , we also test the decrease effect of our method under different |\mathcal{S}| . As Figure 5 illustrates, when |\mathcal{S}| increases from 5 to 10, attacked performances on all LLM retrievers and datasets decrease more in different scales. This observation accords with our motivation on increasing sample population to achieve more optimal solution in non-convex cases. However, when |\mathcal{S}| increases from 10 to 15, some of attacks become less effective. From this phenomenon we infer that there is also a limit in |\mathcal{S}| ’s population effect. When |\mathcal{S}| is larger than the inherent optimal \frac{\eta_{X}}{\eta_{d}} ’s require, attacks become less optimal. Impact of Injected Token Amount (a) (b) (c) Figure 6 : Impact of Injected Token Length We also study the impact of the injected token amount (constrained by the \delta in Eq. 1 ) on our attack and explore its effect when the attacker can afford more injection budgets. As Figure 6 illustrates, however, it’s surprising to find that the attack effects only shift around slightly without significant variations. This result suggests that for the token injection attack on paragraph embedding, a high length of suffix injected tokens length may not have much effect on embedding shifts compared with a fitted length. 5 Related Works Adversarial Attack on Retriever Systems In information retrieval systems, adversarial attack methods generally have two categories: (1) Creating corrupted documents to the corpus and influence the retriever performance on normal documents ( 16 ; 15 ; 19 ; 34 ) ; (2) Manipulating the victim document to decrease or increase its retrieval possibility ( 28 ; 17 ) . With the advance of RAG, adversarial attack against such mechanism also arises attention, which mostly focuses on (3) Manipulating or creating a virus document to be highly likely retrieved and poison the generation process ( 29 ; 13 ; 38 ) , leading to LLM’s unsafe behaviors. Recently some works on agent memory safety also propose methods on (4) Crafting or Manipulating memory entries with poisoning information to be likely retrieved by the LLM retrievers ( 3 ; 4 ) . Token Injection Attack on Large Language Model Injection attacks against LLM is one of the most popular attack type on LLM robustness study. Technically these attack could be summarized into the three categories on crafting methods: (1) Human-crafted attack ( 7 ; 24 ; 23 ) , which is mostly created by human, including amateurs from internet and LLM researchers. (2) Token-optimization-based attack ( 11 ; 8 ; 37 ; 5 ) , which designs specific losses on the LLM downstream task such like generation and retrieval, and trigger specific desired results by heuristically search the optimal combination of injected tokens’ word choices. (3) LLM generation attack ( 32 ; 20 ; 2 ) , which requires a Casual LLM to generate adversarial tokens automatically. Such query-based attack is usually easy to achieve, while also found effective in many cases. 6 Discussion Q: Could this diminishing attack be extended to boost specific document? The success of our proposed attack is mainly relying on "rejecting documents" from its original relevant groups. Therefore on the boosting task, (1) if a manipulated document is hoped to be retrieved with an irrelevant query , (e.g., recommending toy selling websites to queries on a cartoon, indicating a potential child user), our attack is likely to work with adversarial learning loss to be reversed; (2) however, if a manipulated document is hoped to be retrieved by relevant queries with higher rankings , our attack may be not promising since it aims for "out-of-group" relevance rather than "within" relevance. Q: Is there defense method for our proposed attack? Existing defense methods on retriever systems mainly focus on corpus poisoning attack ( 9 ; 27 ) , i.e., one or several adversarial documents are injected into the corpus, and get boosted into the retrieved list. This setting naturally leads these works to focus on detecting poisoned retrieved documents ( 21 ; 35 ) , or ensuring robust RAG generation ( 29 ) , however, leaving the documents not retrieved in the shadow. This post-retrieval characteristic makes these attacks naturally not applicable against our decreasing attack. In all, to our best knowledge, there’s no defense method applicable for our attack so far. 7 Conclusion In this paper we study the vulnerability of LLM-based retrieval by proposing a query-agnostic black-box attack, which requires no knowledge of victim model parameters, access nor the target victim query, and stays effective with zero-shot transferability against different retrievers at the same time. To achieve our attack, we first establish a theoretical framework to describe LLM retrievers’ property and verify it empirically. Then based on this framework, we analyze the transferability of the surrogate model attack, and design an adversarial learning mechanism with a word-embedding-surrogate strategy. Our proposed attack is validated to be effective against most popular LLMR models on challenging benchmark datasets. Through this work, we further call for defense methods that not only focus on post-retrieval detections but also build robust retrieval itself. Impact Statement This paper studies the robustness of LLM-based retrieval systems by identifying a previously underexplored failure mode under realistic black-box conditions. The primary goal of this work is to advance the understanding of retrieval behavior in modern LLM-based systems. The techniques analyzed in this paper are intended to characterize system vulnerabilities rather than to promote adversarial misuse. Similar retrieval failures may arise unintentionally in practice due to benign document edits, formatting changes, or content updates. By systematically studying such behaviors, this work aims to inform the development and evaluation of more robust retrieval systems. This research does not involve human subjects, personal data, or user interaction logs. All experiments are conducted on public benchmark datasets using open-source models. We do not identify additional societal consequences beyond those commonly associated with research on system robustness and security in machine learning. Appendix A Prompt Designs A.1 Prompts for Theoretical Verification To verify the rationality of the \epsilon - p_{\epsilon} -Precise retriever definition, we utilize the following prompts to generate 240 knowledge documents in 24 different topics: topics = ["Chinese Politics","Japanese Politics","American Politics","English Politics", "Comedy Movies","Action Movies","Horror Movies","Romantic Movies", "Electronic Engineering","Computer Engineering","Biology Engineering","Chemical Engineering", "Gothic Architecture","Baroque Architecture", "Modern Architecture","Renaissance Architecture", "Soccer","Basket","Baseball","Golf", "American Geography","Asian Geography","European Geography","African Geography"] for topic in topics: seed = randint(0,1000) Prompt messages = [ {"role": "system", "content": "You are a helpful assistant. A topic with a random seed is given, please write a knowledge document within the topic."}, {"role": "user", "content": f"topic:{topic},seed:{seed}"} ] A.2 Prompts for Generating the Initial Queries To enhance the diversity of the initial queries while keeping them relevant for the victim documents, we design five different generation prompts and apply them in numeration for the casual-LLM. Content are following: To enhance the diversity of the initial queries while keeping them relevant for the victim documents, we design five different generation prompts and apply them in numeration for the casual-LLM. Content are following: Prompt 1 messages = [ {"role": "system", "content": "You are a helpful query writer for information retrieval system. A knowledge document content is provided followed by the user. Please generate a query with content in which the human asker is facing a practical problem and the document would be helpful to solve the it."}, {"role": "user", "content": doc} ] Prompt 2 messages = [ {"role": "system", "content": "You are a helpful query writer for information retrieval system. A knowledge document content is provided followed by the user. Please generate a query by questioning some factors of the document content."}, {"role": "user", "content": doc} ] Prompt 3 messages = [ {"role": "system", "content": "You are a helpful query writer for information retrieval system. A knowledge document content is provided followed by the user. Please generate a query targeting the document and containing five keywords from the document."}, {"role": "user", "content": doc} ] Prompt 4 messages = [ {"role": "system", "content": "You are a helpful query writer for information retrieval system. A knowledge document content is provided followed by the user. Please generate a one-sentence summary for this document."}, {"role": "user", "content": doc} ] Prompt 5 messages = [ {"role": "system", "content": "You are a helpful query writer for information retrieval system. A knowledge document content is provided followed by the user. Please generate a query that has similar semantics but contains few overlaps with the document."}, {"role": "user", "content": doc} ] A.3 Prompt for Poisoned-RAG The following prompt is what we use for Poisoned-RAG attack: messages = [ {"role": "system", "content": f"The user will followingly input a query and a document. Your task is to generate {num_sts} adversarial tokens following the end of the document and make the information retrieval system to rank the injected document lower for the given query. Please only output the adversarial tokens without any other explanation."}, {"role": "user", "content": f"Query: {query}"}, {"role": "user", "content": f"Document: {doc}"} ] Appendix B Implementation and Dataset Details The datasets we applied from ( 10 ) is illustrated in the following tables: Dataset |\mathcal{Q}| |\mathcal{D}| |\mathcal{D}+| L_{Q} L_{D} Econimics 103 50,220 8.0 181.5 120.2 Psychology 101 52,835 7.3 149.6 118.2 Biology 103 57,359 3.6 115.2 83.6 Robotics 101 61,961 5.5 818.9 121.0 Table 4 : Dataset Description Table 4 describes the query amount, document amount, ground-truth document per query amount, average query length and average document length. In our experiment, we first attack every ground-truth document with injected tokens, and then retrieve the top 25/50 documents among the corpus by the query datas. Then we evaluate the Recall and NDCG based on how many ground-truth documents of the query are retrieved in this top samples and how they rank. Appendix C Verifying \epsilon - p_{\epsilon} -Precise on Jinaai and Gemma Table 5 shows the verification results (see Section 3.2 ) conducted on Embedding-Gemma-300M and Jinaai-Embeddings-v3. The p_{\epsilon} of both are high enough to validate the truthfulness of the definition and deduction we made in Section 3.2 . Topic Name p_{0} p_{0.1} p_{0.2} p_{0.3} In Sim. Out Sim. Engin. Chemical 99.6% 88.3% 87.0% 84.8% [0.66,0.94] [-0.01,0.70] Computer 96.1% 90.9% 87.8% 79.1% [0.67,0.88] [0.01,0.80] Electric 88.7% 86.5% 69.1% 30.0% [0.52,0.90] [-0.00,0.80] Biology 99.1% 90.9% 87.4% 84.8% [0.68,0.93] [0.01,0.70] Movies Action 97.4% 90.4% 29.1% 20% [0.52,0.92] [0.00,0.63] Horror 100% 99.1% 99.1% 89.1% [0.70,0.93] [0.00,0.63] Romantic 100% 96.5% 79.6% 55.2% [0.59,0.91] [0.01,0.58] Comedy 81.7% 53.9% 33.9% 0.0% [0.36,0.87] [-0.05,0.55] Achite. Baroque 100.0% 100% 98.3% 91.7% [0.80,0.95] [0.02,0.64] Gothic 100% 100% 99.6% 92.2% [0.81,0.93] [-0.02,0.62] Renaissance 100% 99.6% 92.6% 87.8% [0.74,0.91] [-0.00,0.64] Modern 100% 100% 100% 89.1% [0.74,0.90] [0.01,0.52] Geogra. American 95.2% 86.5% 77.4% 47.4% [0.56,0.85] [-0.03,0.62] European 96.1% 89.6% 86.1% 53.0% [0.57,0.87] [-0.03,0.65] Asian 97.4% 89.1% 87.0% 78.7% [0.59,0.85] [-0.04,0.66] African 99.6% 89.1% 87.0% 84.8% [0.61,0.86] [-0.05,0.66] Sports Soccer 98.3% 92.6% 78.7% 19.1% [0.50,0.91] [0.01,0.54] Baseball 93.5% 83.5% 50.0% 0.03% [0.47,0.87] [0.04,0.54] Golf 100% 98.3% 87.4% 49.6% [0.60,0.90] [-0.00,0.52] Basketball 100% 83.5% 83.5% 40.4% [0.45,0.78] [-0.05,0.43] Politics American 88.3% 67.8% 37.4% 0.05% [0.47,0.91] [-0.03,0.66] Japanese 87.4% 53.0% 3.9% 36.5% [0.36,0.85] [-0.05,0.66] Chinese 0.07% 0.0% 0.0% 0.0% [0.16,0.92] [0.01,0.66] European 0.0% 0.0% 0.0% 0.0% [0.01,0.87] [0.01,0.65] Topic Name p_{0} p_{0.1} p_{0.2} p_{0.3} In Sim. Out Sim. Engin. Chemical 100% 100% 99.6% 88.3% [0.82,0.95] [-0.02,0.62] Computer 96.1% 97.0% 92.6% 87.8% [0.67,0.88] [0.01,0.80] Electric 100% 97.4% 90.0% 87.4% [0.75,0.94] [-0.09,0.70] Biology 92.2% 87.0% 85.7% 53.5% [0.53,0.90] [-0.02,0.62] Movies Action 100% 100% 97.0% 90.4% [0.73,0.90] [-0.02,0.60] Horror 100% 100% 99.1% 90.4% [0.70,0.93] [0.00,0.63] Romantic 99.6% 91.3% 87.4% 79.1% [0.56,0.92] [-0.05,0.58] Comedy 83.9% 23.0% 0.0% 0.0% [0.25,0.76] [-0.09,0.58] Achite. Baroque 100% 93.5% 91.3% 87.8% [0.75,0.92] [-0.02,0.71] Gothic 99.6% 95.2% 90.9% 86.5% [0.71,0.94] [-0.14,0.71] Renaissance 100% 97.8% 91.3% 87.0% [0.76,0.90] [-0.03,0.71] Modern 100% 100% 90.0% 82.6% [0.65,0.91] [-0.07,0.54] Geogra. American 99.6% 88.7% 81.7% 68.7% [0.61,0.92] [-0.02,0.62] European 99.6% 90.0% 80.0% 50.0% [0.60,0.91] [-0.04,0.60] Asian 100% 96.5% 87.4% 83.5% [0.68,0.93] [-0.02,0.62] African 100% 91.7% 87.8% 80.9% [0.61,0.92] [-0.02,0.62] Sports Soccer 100% 98.3% 94.8% 90.9% [0.69,0.92] [-0.02,0.61] Baseball 95.7% 90.4% 88.7% 63.5% [0.57,0.89] [-0.14,0.61] Golf 100% 99.1% 93.5% 87.0% [0.66,0.88] [-0.05,0.60] Basketball 92.6% 58.7% 16.5% 0.0% [0.36,0.82] [-0.04,0.58] Politics American 0.0% 0.0% 0.0% 0.0% [0.0,0.90] [-0.05,0.61] Japanese 97.4% 89.1% 87.0% 78.3% [0.59,0.91] [-0.02,0.64] Chinese 90.0% 86.5% 73.5% 46.1% [0.51,0.89] [-0.04,0.64] European 100% 93.4% 88.7% 79.1% [0.63,0.89] [-0.01,0.61] Table 5 : 24 context topics’ p_{\epsilon} on Embedding-Gemma-300M (left) and Jinaai-Embeddings-v3 (right) retrievers.